Repository navigation
Conversation
…ord known startup cost HACK-1169: freeze revision 1 of the v5 matched application benchmark before any trials. docs/reference/application-benchmark-protocol.md and its machine-readable copy fix the exact artifacts (v5.0.0-next.5 at 28de2f8 and stable 4.1.1, 4.2.0, 4.2.1 with release asset SHA-256 digests), the M3 host with Docker CLI 29.4.0, Compose v5.1.2 and OrbStack 2.2.3, Event Agent plus a pinned small fixture, capacities, cache classes, a 10-round Williams rotation, p95 sample floors, observer overhead accounting and the program budgets. scripts/check-application-benchmark-protocol.py validates the frozen file and judges retained summaries offline with closed, value-free codes; anything short of complete, sufficient samples is inconclusive. Offline unittest controls run in CI. docs/performance.md records the user-accepted known v5 limit (HACK-1123): the storage-witness cost of a warm native up after #313, with rationale and revisit trigger. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…nings
HACK-1169 rework of the application benchmark protocol checker.
Evaluation: the candidate can never be not_applicable (exit 2,
summary_candidate_not_applicable). A not_applicable record is exactly
{not_applicable: true, reason} and cannot carry samples. A comparative budget
whose baselines are all not_applicable gets an inconclusive_no_baseline
verdict, so it is never accepted. Only warm, single-instance summaries are
judged. Samples must be non-negative, expected a positive count, and flagged
and failed lifecycles are accounted: any flag, unaccounted admission or failed
baseline lifecycle is inconclusive_admission, and a failed candidate
lifecycle fails. Every malformed summary or protocol value maps to a closed
code instead of a traceback.
Protocol validation now pins each budget's statistic, workload scope (only
the watcher budget is limited to the small fixture, as in Gate C; idle CPU
and cached status now apply to both workloads), the observer factor of 2,
p95 method and confidence, rounds_per_workload, the Williams design, workload
shapes, the required baselines in every workload, the Compose lane against
the frozen host, the Gate C phases (adding restore) and the 1/8/32
branch-instance and 1/32 MCP-client scale points.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The protocol validator checked only the format of artifact commits and digests, so a different well-formed candidate commit, a removed asset, a recast lane kind or backend, or a changed candidate capacity still validated. Pin the revision-1 lane artifacts, lane kinds and backends, candidate capacity, host values, workload pins and correctness checks exactly, with closed refusal codes. Bound every protocol and summary number to the interoperable JSON range so an oversized integer is refused as malformed (exit 2) instead of raising OverflowError, and keep an overflowing ratio a failure without printing a non-standard JSON constant. Document why revision 1 freezes the native development profile although the candidate labels it experimental-development-not-benchmark-qualified: graph allocation refuses every other profile and native up always requests it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The validator still accepted a different balanced Williams order, so reordering the lane list and regenerating the schedule validated while the published order table and --schedule disagreed. It also judged sample counts against the program floors (5 runs, 59 tail samples) rather than the frozen revision-1 values (10 and 60), so a protocol edit could lower them without refusal. Pin the revision-1 order exactly (schedule_order_mismatch) and treat the frozen sample counts as floors. Check that the published order table matches the frozen order round for round, and that the page states both frozen sample counts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…members unready until the readiness deadline The startup Compose readiness check refused two Docker on-failure restart races that #350 left open: - Case 2 (captured in run 39436132066, after-compose observation 3): the retry container was inspected running, then exited, and Docker released its endpoint for the restart backoff before the same scan read the bridge. The member was absent with no placeholder, so startup refused with live-endpoint-membership. - Case 1: the retry container was an ID member in the first scan, then restarted, and the recheck read its new ep- placeholder. Startup refused with network-member-id. Startup now keeps both unready, but only for a selected, non-oneoff on-failure container read running or restarting, on the single owned default bridge. A recheck placeholder is admitted only for the same container that the first scan saw by ID or as absent. A placeholder that persists across both scans, and every foreign, malformed, strict or saved shape, still refuses as before. waitNativeComposeStartupOwned bounds this with the existing readiness deadline. If the deadline passes while the last observation was unready only because of these running shapes, one settled check refuses a member that is still missing or placeholdered. Refs HACK-1279 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Integration check only — do not merge this PR. CI runs here on the combined result of merge train b20 before each PR is squashed into
nextindividually.train/b20=next8c79000 plus #362 e649d42 (G11, HACK-1279 restart races) and #334 b5c2e9f (F12, HACK-1169 benchmark protocol). Both passed their independent verifiers and their own CI (16/16).Local checks on the combined branch: typecheck clean,
check0 errors, 96 ownership/trace/source-check tests pass (Bun 1.4.2), benchmark protocol valid and its 37 unit tests pass.When this is green, both are squashed in order with
--match-head-commitand next's final tree is checked against 99ac0d13. Then this PR is closed.🤖 Generated with Claude Code
Need help on this PR? Tag
@codesmith-botwith what you need.