Skip to content

Playgrounds: quick figures and launch latency eval - #508

Merged
brsbl merged 17 commits into
mainfrom
bb/playgrounds-launch-eval-and-quick-figures-thr_vv8zgxmczt
Oct 20, 2026
Merged

brsbl merged 17 commits into
mainfrom
bb/playgrounds-launch-eval-and-quick-figures-thr_vv8zgxmczt

Conversation

@brsbl

@brsbl brsbl commented Oct 19, 2026 •

Copy link
Copy Markdown
Owner

Ordinary explanations inherited the full Figure workflow, and Personal threads could not publish files from their own storage. Explanations now default to one small SVG mechanism with visible progress and no clarification; explicit deeper requests retain the full workflow. File publication accepts the invoking thread’s own storage while preserving host-enforced path confinement.

Adds a 20-prompt launch eval on both default providers in fresh non-Git Personal threads. It measures send→visible card, questions, progress and real control input. Missing cards score at 15 minutes; the fixed 30-second render grace never extends that cap. Saved-card rescoring and visual adjudication preserve all original agent runs and timestamps.

Evaluation Median / p90 (min) Cards with every control live Questions
baseline / claude-code 9.93 / 15.00 14/20 0
baseline / codex 1.91 / 2.75 6/20 0
prototype / claude-code 3.93 / 5.52 19/20 0
prototype / codex 1.46 / 1.70 15/20 0

The launch bar is not met; gate 1 was accepted because both prototype medians are under six minutes. The gate-1 report contains every failing baseline/pilot control, corrected verdicts, native-disclosure supplements and Claude phase timing. Shared-host pressure and the prototype’s inclusion of R2 #510 prevent attributing the whole improvement to quick mode.

Validation:

  • Remote CI is required on final head 0e9b5cb8a979ac9396ab46d410f79527e1e9ba1a; its current status is shown by the PR checks.
  • Full cumulative Slop Cop completed architecture, correctness, security and performance panels; P1 fixes received targeted review. Historical browser regressions cover ten proof fixtures, and final-head E2E adds the native-disclosure regression. P2 follow-ups: tie semantic proof to the exact changed visible element, and handle newly revealed output regions.
  • Final-head isolated E2E: PASS. Personal publication boundaries, both default providers, quick/deeper routing and actual control operation are covered.
  • Fresh default-provider doorbell smokes: Claude 236.298 s, Codex 105.166 s; visible progress, zero questions, real press/release and native disclosure verified at 390 px. Ten publication allow/deny/recovery checks passed. Deeper request selected the full Figure route; full Figure completion was outside this bounded pass. Generated-card P2 follow-ups: secondary text token collision and boolean parameter normalization; retained for the gate-2 PR.
  • No visual change to the plugin UI components; generated evaluation cards and their before/after captures are evidence, not fixed product screenshots.

Closes #511

BB-Thread-ID: thr_vv8zgxmczt

AGENT GENERATED

@brsbl
brsbl marked this pull request as ready for review October 20, 2026 07:31
@brsbl
brsbl merged commit 7be5770 into main Oct 20, 2026
42 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Playgrounds: launch eval and quick figures in Personal threads

1 participant