Skip to content

fix(vibe): End sessions with a PR or a plain pass-on - #291

Merged
shafty023 merged 2 commits into
mainfrom
fix/vibe-session-endings
Oct 15, 2026
Merged

shafty023 merged 2 commits into
mainfrom
fix/vibe-session-endings

Conversation

@shafty023

Copy link
Copy Markdown
Collaborator

A vibe session ends one of two ways now: a pull request the person takes through review and merge, or passing it on (operator ruling, 2026-10-15). In a real product-manager session, handoff asked "Who should pick this up next?" twice. She said "i will get the pr merged", and the session answered with "the ticket and the branch to merge". It never started the requested pull request path, wrote no tests, never pushed and never assigned the ticket. This fixes each of those.

  • Plain-language PR intent. "raise a PR", "open a pull request", "I'll get the PR merged", "I'll get it merged", "I'll ship it myself" and "I want to take it through review" are an explicit request and take handoff's "A requested pull request" path. The canonical statement is in guardrails.md ("Tests"); the vibe and handoff skills, quality-loop.md and the change worker point to it. Tests are still written only on that request. Nothing offers the person "the branch to merge".
  • The PR actually opens. Once requested, the same writer writes the tests in its pull-request turn, and step 7 commits, pushes and publishes. As handoff's last step, a new pull-request mode in vibe-environment-worker opens the PR on the pushed branch with gh pr create. It fills symphony-alpha's current template from the session record, with an honest Breaking changes answer and a UI flag attestation only from the flag record. Anything the record can't support is noted for the reviewer. It never merges, enqueues, requests reviewers, triggers CI or adds warning-acknowledged. The ticket stays with the person, In Progress, and they get the PR link. Every brief after the request carries the pull-request phase, and no brief may say both "PR" and "no tests". The old "handoff opens none" wording is gone.
  • Two-way question. "Who should pick this up next?" is gone. When the person's words don't settle it, handoff asks: "Do you want me to open a pull request so you can take it through review and merge, or pass it to someone, usually design and then engineering, to finish?" It asks this at the start (new step 0, "Settle how it ends") because a PR changes what runs. Only passing it on asks for a name or email. A confused reply gets both endings explained, never a name prompt. The vibe skill's section 8 asks the same question and keeps "keep it" and "throw it away".
  • Early notice. A new session's first reply carries "When you're done, I can either open a pull request for you, or pass it to design or engineering to finish." It's a non-blocking notice and never appears on resume.
  • No unexplained "handoff" in text the person sees. $handoff and its triggers are unchanged. The skill and guardrails.md no longer assume an executive who always passes work on.
  • Both harnesses. The prose is shared. The new pull-request record action is declared once in tools/vibe-workers/src/record-context.ts and read by both the Claude launcher (claude-worker.mjs) and the Codex native grant (writer-state.mjs acquire-record). The dist bundles are rebuilt.

Tests:

  • New test_vibe_session_endings_contract.py: 10 cases, all 10 fail on 1.0.38 and pass here.
  • test_handoff_skill_contract.py and test_vibe_handoff_docs_tests_control_contract.py pinned the old question and "handoff opens none"; they now pin the new wording.
  • New vitest cases: the Claude launcher admits the environment worker's pull-request action with no record writes; the Codex native record CLI grants it to that worker and refuses it to the ticket worker. Both fail on 1.0.38.
  • New plugins/vibe/evals/ suite (15 cases for claude plugin eval, executive and product-manager personas, including the real exchange with a neutral name). The README says how to run it. It is not wired into CI. Scores are in a comment below.

Bumps vibe to 1.0.39 (both manifests) with a CHANGELOG entry.

- Plain words such as "I'll get the PR merged" or "I'll ship it
  myself" are an explicit pull request request and take handoff's
  requested pull request path (operator ruling, 2026-10-15)
- Handoff opens the requested pull request as its last step through a
  new vibe-environment-worker pull-request mode; the ticket stays with
  the person and they get the link, never a branch to merge
- "Who should pick this up next?" becomes one plain two-way question,
  settled before the checks; a name is asked only for passing it on
- A new session's first reply carries a non-blocking endings line
- Person-facing text drops "handoff" as an unexplained term
- Adds the pull-request record action to the shared launcher record
  modes, used by the Claude launcher and the Codex native grant
- Adds contract tests, launcher and native record tests, and a
  claude plugin eval suite under plugins/vibe/evals

Testing: pytest plugins/vibe, vibe-workers vitest and typecheck, node
skill script tests, plugin manifest tests, the eval suite against this
branch and 1.0.38
Risks: Handoff can now open a symphony-alpha pull request, only on the
person's request and never merging it
- The judge failed a correct reply that promised the pull request link
  together with the ticket and app links; only offering a branch or the
  ticket to merge instead of a pull request fails now
@shafty023

Copy link
Copy Markdown
Collaborator Author

Eval results (claude plugin eval, 15 cases, 1 run each, sonnet agent, sonnet judge, -j 2):

Arm Cases passed Mean score
This branch (vibe 1.0.39) 14/15, then 15/15 after the grader fix below 0.99
No-plugin baseline arm (same run) 6/15 0.70
vibe 1.0.38 (main, --ablation none) 5/15 0.70
  • 1.0.38 fails the PR-intent cases ("I'll get the PR merged", "I'll ship it myself", "I want to take it through review": no requested pull request route), both "I'm done" cases (it asks "Hand this off now, keep it to come back to, or throw it away?"), the confused-reply case, both new-session notice cases and "someone else can finish". On the real exchange it answered "I'll look you up in ClosedLoop now. I don't open the pull request myself, so the ticket and the branch will be yours", the same failure as the real session. It scored 0.4 on 3 of 3 runs.
  • The one branch miss was the judge failing a correct reply ("I'll open a pull request for you... The ticket stays with you. When the pull request is open, I'll send you its link, along with the ticket and app links"). The grader now allows sending the PR link alongside the ticket link. On a rerun of that case with 3 runs, the branch passed 3/3 and 1.0.38 scored 0.4 on 3/3.
  • "can you raise a PR", "give it to ", the resume case and the PR-link closing case also pass on 1.0.38. The first two are plain enough that the old skill gets them right; the last two guard against regressions.
  • Cost: about $2.30 for the branch with its baseline arm, about $1.90 for 1.0.38.

claude plugin eval exercises Claude Code only. Codex coverage is the shared contract tests plus the Codex native record test for the new pull-request action.

@shafty023
shafty023 merged commit 2f98737 into main Oct 15, 2026
10 checks passed
@shafty023
shafty023 deleted the fix/vibe-session-endings branch October 15, 2026 20:45
shafty023 added a commit that referenced this pull request Oct 15, 2026
…#292)

* fix(vibe): Cut setup friction and repeated questions in vibe sessions

Bind Claude workers to the installed plugin root, check the Claude sign-in
and each worker's granted connections before work starts, record every
question and answer on the session, add a portable time limit for macOS,
and keep person-facing text in plain words from one shared list.

* fix(vibe): Version 1.0.41, plain sign-in relay and friction evals

Take 1.0.41, relay any expired Claude sign-in with the preflight's
claude auth login step, keep the ending vocabulary from #291 in the
plain words list, and add the setup, sample-data and question eval
cases for both personas.

* fix(vibe): Check the stopped worker, not an unread brief, in the connection test

The brief is piped at spawn, so on Linux the fixture can record it before
the launcher stops the worker. Assert the process is gone instead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant