Repository navigation
harness: add bounded VATS/GVO runtime and benchmark adapters - #488
amy-why-3459 wants to merge 5 commits into
Conversation
Separate editable lineage from candidate-specific verification evidence. Retain scoped attempts and dependency epochs, select complete development candidates, and enumerate bounded completion plans with shared preparation and cost caps. Cover failed-parent repair, stale evidence, deterministic selection and planner feasibility with CPU controls; document the shared execution integration boundary. Agent-assisted-by: Codex (implementation, tests and documentation) Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent non-voting review: not approved; acceptance and execution evidence are incomplete.
Reviewed head 853c92f35f3b4d3df841958ba33568f690f86c5d against base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7, on the other-author draft PR. I inspected both new executable modules, the synthetic fixture and both new test files, plus machine CI evidence. AGENTS and repository review guidance were checked. No other reviews, PR result narrative or mixed Issue discussion informed this assessment.
Code facts established by static inspection
- Search evidence is bound to the complete registered Attempt, including candidate/artifact, context, check, execution key, reservation and lease identity. Receipt registration rejects changed bindings/conflicting duplicate receipts; evidence selects the latest attempt and excludes invalidated dependencies and INCONCLUSIVE outcomes. Acceptance requires every required check to pass, including the objective, and preserves the qualified root on a score tie.
- Candidate edit ancestry is distinct from generation/evidence dependencies: a fully checked repaired child can be selected despite a failed or incomplete edit parent, while inherited dependency invalidation removes affected descendants from admissible selection. This is a substantive semantic distinction from treating every parent failure as automatic descendant failure. It should not be assumed to settle the older bounded-successor RFC's acceptance contract.
- Completion planning constructs prerequisite closures, counts shared jobs once, applies cost/job/attempt bounds and conflicts, rejects outstanding unsettled work, and returns search identity/state revision without spending the supplied budget. A plan is not an execution reservation or proof that its costs were incurred.
- The inspected tests cover wrong receipt bindings, late results after invalidation, same-byte distinct candidates, immutable identities, fresh retries after INCONCLUSIVE, bounded fallback, and a three-candidate exhaustive planning oracle. The engine/skill-labelled fixture uses retained files containing synthetic numbers; it is not two real adapter integrations, independent holdout qualification or a performance experiment.
I did not identify a confirmed actionable defect in this bounded static pass. One useful additional coverage case is cross-candidate prerequisite closure: selecting one completion seed can incidentally complete another candidate. The current oracle fixture shares setup but does not exercise that coupling. Keep the seed-enumeration bound distinct from a bound on all incidentally completed candidates; this is a coverage/contract clarification, not a demonstrated runtime defect.
Important limits: SearchState revision checks operate on the supplied in-memory state. These modules do not themselves implement durable compare-and-swap, crash recovery, actual reservation/lease authorization, cancellation, process cleanup, holdout access control or atomic budget accounting. Nonempty reservation/lease identifiers are identity bindings, not proof those resources were reserved. The PR's isolated scope is offline candidate trees and verification planning; absence of a runner here is not by itself a code defect, but these helpers must not be presented as completion of those wider operational guarantees.
Evidence actually inspected
- lint-and-smoke 134586515319: exact head appears in the raw log; Ruff passes and 8 passed in 0.05s. Those smoke tests do not establish the new search-tree/planning tests.
- installed-package 3.10 134586617451: raw successful sdist/wheel build at synthetic merge
b92eff795921472b952a4e8e0401cffce4a866e0of this head into the stated base. - At inspection, full-check 3.10/3.11 and Windows were still running, macOS queued; browser was skipped. Installed-package 3.11, references and declarations had success metadata. Completed full-check logs were not available to verify the new tests.
No repository tests or code were executed by me. The detailed current AC remains unavailable in a safely isolated form; the earlier #363 requirements are not automatically the full edited VATS/GVO contract. Changed documentation was not used as acceptance authority or as proof of implementation. This COMMENT records a limited independent code/evidence review, not whole-RFC acceptance, workflow efficacy, permission for pilots, code changes or merge.
david6666666
left a comment
There was a problem hiding this comment.
One numeric boundary to address in the offline search helpers.
| def verified_gain(state): | ||
| selected = accept(state) | ||
| check = state.contract.objective_check | ||
| gain = (evidence(state, selected.candidate_id)[check].objective |
There was a problem hiding this comment.
Both receipts can have finite objectives while their difference overflows. With a root of -1e308 and a selected candidate of 1e308, verified_gain() returns inf, which downstream strict JSON cannot serialize. Could we reject a non-finite gain explicitly (or bound the objective domain) and add a regression for both objective directions?
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent non-voting CI follow-up to review 5951024572: completed regression evidence, but not whole-scope approval.
Head 853c92f35f3b4d3df841958ba33568f690f86c5d, base 8aba657da79e8ed9f6e6fbb33437e9d9fdd8f1c7. I independently retrieved the newly completed machine logs rather than treating success metadata or author statements as test evidence.
Both full-check logs in Ready run 44240402818 show checkout b92eff795921472b952a4e8e0401cffce4a866e0, explicitly merging this exact head into this base:
- Python 3.10 / 134586617402: Ruff passes; 5149 passed, 36 skipped, 671 subtests passed, 1 warning, 474.66s.
- Python 3.11 / 134586617133: Ruff passes; 5149 passed, 36 skipped, 671 subtests passed, 1 warning, 510.69s.
These invoke pytest -rs across the configured tests directory. I rechecked the current test_speculative_search.py and test_verification_plans.py: 19 and 10 test functions before parameter expansion, with no skip decorators found; neither appears in the complete skip summaries. This is aggregate execution evidence for the newly inspected offline tests, not an individually verbose PASS list or an independent local rerun. The 36 skips include optional dependencies, real Cannbot qualification and hardware/platform-specific cases; they are not 36 passes.
Raw platform logs at the same merge report Windows 8 passed + 25 passed, and macOS 8 passed. Their selected modules concern plugin/DSH and Windows portability/session behavior, not a full cross-platform execution of the new planner/tree tests. Dashboard-browser is skipped, not passed. Other Ready jobs are now successful by metadata. This resolves the pending CI evidence noted previously.
The isolated specification clarification confirms max_targets limits the union of completion seeds, not the number of candidates incidentally completed by prerequisite closure. Consequently I do not classify extra incidental completions as a cap violation; my prior cross-candidate prerequisite suggestion remains additional coverage only.
The offline code assessment is unchanged. Receipt identity/dependency bookkeeping and proposal generation do not establish durable CAS/recovery, authenticated reservations/leases, real cancellation, holdout isolation or actual Engine/Skill adapter efficacy. Detailed current RFC acceptance provenance remains incomplete; old bounded-successor requirements must not silently substitute for the edited VATS/GVO scope. No code/test execution, campaign restart, model/device/paid experiment, code change or merge was performed by me.
Source accounting: isolated metadata for inline4609449850@2026-11-18T05:09:44Z contains no direct owner mention or independent machine locator. Its inseparable technical narrative was not consumed or adjudicated. This follow-up is driven by the independently verified CI transition, not that source's opinion.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review — NOT APPROVED. COMMENT rather than a Request changes vote.
Range: previous reviewed head 853c92f → abe72f6 → current 5ee4181; base remains 8aba657. Previous own review: #488 (review) . This is a substantial expansion from offline primitives into persisted VATS execution, CPU GVO roles, continuation/final control and Engine/Skill adapters, not merely the earlier passing offline tests.
P2 — Persisted not-started expiry strands Search recovery. In harness/vats.py:391–421, _execute atomically writes search.pending (and registers a verification Attempt) with its claim, then calls WorkControl.launch. In work_control.py:171–188, expiry between claim and launch conclusively records settled/not_started and releases the domain, but leaves search.pending untouched. Since the adapter never ran, vats.py:356–364 cannot find its receipt and permanently reports Unknown execution; finish at 635–640 also rejects pending work. A scheduling pause or approaching absolute deadline can therefore prevent a search with a valid qualified root from completing normally even though execution is known not to have started. Reconcile the durable not_started terminal record with the pending Attempt/work-block state, without relaunching uncertain work. Add a Search-level claim→expiry→recover/limit-finish test; the existing WorkControl-only expiry test does not exercise this composition.
P2 — Known overspend is ignored while cleanup is unknown. work_control.py:213–217 takes the unknown branch whenever cleanup_verified is false, even when actual_ms is known and exceeds the reservation. available() at 103–105 subtracts only the old cap, and another domain can still claim. Static example: campaign cap 1000, final reserve 100, claimed cap 200, then settle(actual_ms=950, cleanup_verified=False). spent remains zero, blocked remains false and available reports 700, allowing another 700 commitment despite known spend already consuming the final reserve. Retain the domain/unknown-work restrictions, but conservatively account for the known expenditure lower bound or block on known overspend; reconcile later without double charging. Add the combined known-overrun + unknown-cleanup + second-domain claim regression. This is source-path reasoning, not a locally executed reproducer.
Other inspected changes / prior findings: the latest Engine post-execution frozen-resource recheck and expanded Skill controller identity are present; their new mutation test is a source assertion, not independently executed evidence here. GVO separates forced continuation scorer targets from actor groups, verifies retained training labels, and uses retained checkpoints for Reset/Carry recovery. I did not establish an additional concrete training-formula defect in this bounded pass. CPU update timing is measured after update returns; block-on-overrun is not preemptive termination. Native host/process cleanup is not filesystem isolation or a GPU scheduler reservation. The previous completion-seed clarification remains unchanged and is not reopened.
Evidence: exact-head lint-and-smoke job 134828062677 raw log identifies 5ee4181, Ruff success and 8 passed in 0.04s for test_smoke.py/test_claim.py. That does not execute the new VATS/GVO tests. Current full-check 3.10/3.11 and Windows are in progress, macOS queued, dashboard-browser skipped; package/reference/declaration checks have success metadata. Cancelled intermediate-head jobs and historical 5149-test results are not current-head passes.
Before approval, address the two control-state/accounting paths and demonstrate their regression tests plus applicable new-head full integration checks. Current Issue #363 normative sections were isolated from opinions: its proposed recovery/budget/final/dual-pilot requirements are not evidence of maintainer acceptance or authorization for real experiments. Synthetic CPU fixtures, finite-edit softmax roles and a comparator bridge do not establish both real Engine/Skill VATS/GVO/final pilots or performance gains. Whole-RFC acceptance provenance remains unresolved.
Performed: actual incremental code/test inspection across execution/accounting, adapters/CLI, roles/GVO/continuation/final; current AGENTS, isolated normative requirements, own-review deduplication and machine CI evidence. Not performed: repository-code or test execution, model/device/provider trials, paid experiments, edits, CI rerun or merge. No other reviewers' opinions were read or used. Optional follow-up: keep explicit tests distinguishing known not-started work, unknown launched work and known overspend with unknown cleanup, rather than testing these states only in isolation.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: 5ee4181 → bd2bfc4; base remains 8aba657. Previous own review: #488 (review) . COMMENT, still not approved.
The four-file delta adds an initial_conditions digest to Search.start and requires equality across GVO examples. It includes candidate/provenance, dependency epochs, actual execution/reservation/lease identities and receipt evidence, excluding only receiving attempt_id/search_id. I cross-checked import_qualified_root: its receiving-scope rebinding changes those two excluded fields, so independent searches can still share the same qualified evidence without falsely differing only by their local IDs. The added assertion rejects a differing digest. The documentation clarifies use of Comparator.holdout_scope, not an executable final-gate change. No new confirmed source defect was identified in this narrow delta.
Neither prior P2 is resolved by this commit. work_control.py is the identical blob, and the relevant VATS claim/launch/recover paths are unchanged:
- A conclusively not-started expired launch still leaves search.pending (and, for verification, its Attempt) behind. Receipt-only recovery cannot obtain a receipt from an adapter that never ran; normal finish remains blocked. Coordinate the durable not_started outcome with search/work-block state and add the composed expiry/recovery regression.
- Known actual_ms above cap with unknown cleanup still takes the unknown branch without blocking or charging the known overrun; available() retains only the original reservation. Other domains can still claim against an overstated available budget. Preserve resource exclusion while conservatively accounting/blocking the known overspend, with an idempotent reconciliation regression.
These remain static code-path findings, not executed failures. The new initial-condition equality does not address either state/accounting path.
Current evidence: raw lint-and-smoke job 134831307114 identifies this revision, Ruff success and 8 passed in 0.07s for smoke/claim tests. At inspection the full-check, host and package jobs were still in progress/queued; dashboard-browser was skipped. No completed current-head new-runtime regression result was established. Prior-head aggregate passes are not relabeled as this head's results.
Before approval: resolve the two P2 paths, execute the targeted combined-state regressions and applicable full checks, and retain the previously stated distinction between CPU/synthetic controls, proposed Issue #363 acceptance and real Engine/Skill pilots. The earlier acceptance-provenance and real-pilot gaps are not cleared here.
Non-blocking coverage suggestions: construct same-root-byte groups with changed actual receipt evidence/epochs to test digest derivation, plus same evidence with different receiving IDs to test admissibility. For compatibility with previously persisted experimental searches, define an explicit version/migration policy: records predating initial_conditions cannot be consumed by direct dictionary indexing without that field; do not fabricate their historical starting evidence.
Performed: all four changed files, relevant root-import and unchanged failure paths, own-review/head checks and machine smoke evidence. Not performed: repository-code/test execution, real model/device/provider trials, edits, CI rerun or merge. No other reviewers' opinions were read or used.
Signed-off-by: amy-why-3459 <wuhaiyan17@huawei.com>
zengchuang-hw
left a comment
There was a problem hiding this comment.
Independent incremental review: bd2bfc4 → 96e3c50; unchanged base 8aba657. Previous own checkpoint: #488 (review) . COMMENT, not approval.
I inspected all seven changed files. The delta removes direct Model imports from the new Harness modules and dispatches through the injected snapshot operations. For the supplied CPU Snapshot, choose/predict wrap the same choose/dot functions; train reconstructs Choice with the same tuple conversions previously performed in gvo, then passes the same reference/actors/values/pairs/settings to update; restore constructs the checkpoint's Snapshot type. The Harness finite-return validation remains equivalent for recorded numeric observations. I found no new concrete parameter/serialization defect in this bounded refactor. The documented dependency direction is improved; this is not an executed integration result.
The two previous P2 findings remain unresolved. I rechecked the current failure paths rather than relying on discussion state:
- work_control.py is still blob 3a0993c42845765146573e31d4bf1a1ce3e60628. Expired launch persists settled/not_started without reconciling VATS pending/Attempt state; recover still requires an adapter receipt that cannot exist for unstarted work, and finish still rejects pending work.
- The unknown-cleanup branch still ignores a known actual_ms overrun for blocking/available-budget purposes, while available subtracts only the old reservation. The previously described 1000-cap/100-final-reserve/200-claim/950-known-use sequence can still expose 700 available to another domain. Resource exclusion alone does not make this budget view conservative.
Before approval, fix those two composed-state/accounting paths and supply their targeted regression execution plus the applicable current-head complete checks. This dependency refactor does not resolve them. Earlier initial-evidence consistency and frozen-resource improvements remain present; no formerly resolved item is reopened.
Machine evidence inspected: exact-head lint-and-smoke job 134833551072 raw log identifies 96e3c50, Ruff success and 8 passed in 0.06s for tests/test_smoke.py and tests/test_claim.py. At inspection full checks, Windows/package/reference jobs were running, macOS queued and dashboard-browser skipped. This does not establish completion of the new search/GVO runtime tests. Historical-head results are not current-head passes.
Optional coverage: add direct injected-role boundary tests for the round-trip Choice document, checkpoint restore and same-input predict/train behavior, so future non-CPU role implementations cannot silently diverge at this seam.
Performed: seven-file static delta review, current original failure-path reread, own-review/head deduplication and raw CI evidence. Not performed: repository-code/test execution, real model/device/provider pilots, edits, CI reruns or merge. Prior proposed Issue #363 acceptance provenance and real-pilot boundaries remain unchanged. No other reviewers' opinions were read or used.
Problem and resulting behavior
The serial lifecycle cannot edit incomplete/failed candidates, choose bounded verification bundles, or train search roles from independent search returns. This PR adds a bounded synchronous VATS/GVO runtime: retain real edits, verify selected candidates, atomically settle evidence and costs, update separate CPU Drafter/Policy/Scorer parameters, and search again with the new frozen checkpoint. A repaired child can win independently of its parent's quality outcome, but only with its own complete evidence.
Refs #363
The user expanded the original offline slice to the full proposal and requested repository benchmarks. This remains a draft implementation toward that scope: the supported local profile and the outstanding acceptance requirements are documented in
docs/speculative-search.md. The complete RFC is not claimed complete.Implementation
inferrsi search engine/status/exportand documented Python composition. Existing serial session behavior stays unchanged.Validation
Revision:
96e3c50c54a9c30581492b8dc86f916e5c5fec43.ruff check src tests tools: passed.git diff --cached --check: passed before commit.search engine --help: passed.qwen21-denoise-npubenchmark: one CPUsmoke/check-onlyinvocation with an actual mathematically equivalent candidate edit; baseline and candidate correctness passed.speedupis null. This is mathematical correctness evidence, not NPU performance or full-task qualification.abe72f6; later refinements have their focused reruns above.test_knowledge_compatibility_shells,test_legacy_run_forwarder_gateandtest_target_module_import_gate: Harness directly reached Model. Commit96e3c50removes those imports through injected role operations; all three full modules pass in the 152-test rerun. The historical full run remains failed; it is not relabeled as passing. Final-head GitHub full regression is pending.The process-backed numeric grader and synthetic Skill report controls are software tests, not production Engine/Skill pilot evidence. No provider/model API or GPU/NPU performance experiment was run.
Remaining full-issue requirements
The supported profile is a trusted local host, one worker and small bounded menus. Domain exclusion applies only to cooperating controllers sharing the same RunStore; an arbitrary domain name is not a GPU/scheduler allocation or OS sandbox. Arbitrary standalone shared-setup jobs are represented by the offline planner but runtime adapters package setup into whole checks.
Production resource qualification, hard preemption of in-process callbacks, preflight/setup lifecycle accounting, campaign-wide provider token/currency reservations and multi-worker comparisons remain outstanding. Skill per-comparison bounds do not establish a global token/currency budget. The supplied operator model does not provide an LLM training adapter or supervised warmup data.
Real Engine and Agent/Skill VATS/GVO/final pilots and the AC-10 serial/VATS, bundle, learning, Reset/Carry and worker comparisons are not run. This host has CUDA devices but no NPU environment for the packaged full-performance task; a controlled live Agent/provider configuration and its budget have not been supplied. Separate independent qualification is also required before promoting a learned checkpoint as a default improver.
The AC-01–AC-10 support/remaining-evidence matrix is in the documentation. No parent-issue closure, production acceptance, independent review, efficacy gain or merge approval is claimed.