Skip to content

test(core): fix the stopped-command race and report stalled PowerShell calls - #491

Merged
shuv1337 merged 4 commits into
integration-v2from
shell-pwsh-timeout
Nov 13, 2026
Merged

shuv1337 merged 4 commits into
integration-v2from
shell-pwsh-timeout

Conversation

@shuv1337

@shuv1337 shuv1337 commented Nov 13, 2026 •

Copy link
Copy Markdown
Owner

Summary

This PR fixes two timing-sensitive cases in packages/core/test/tool-shell.test.ts that have each failed the unit gate on a GitHub runner.

  1. ShellTool stopped > stops a command the user stopped, keeps its file… (failed on chore(ci): run the full suite on sync and release PRs #490, unit attempt 1, with the killed command's output missing its second line). This is a test race, and it's fixed. stopCall cancelled as soon as any output existed. bash makes stdout line-buffered, so printf 'first\nsecond\n' reaches the pipe as two write calls (strace: write(1, "first\n", 6), then write(1, "second\n", 7)). A cancel that landed between them killed the command before second was written. stopCall now takes the number of bytes to wait for, and this test waits for all 13 before cancelling. The other callers keep their behavior: 1 byte, or 0 for the commands that print nothing.
  2. ShellTool ordinary shell syntax > PowerShell: ordinary syntax reuses approvals: Write-Output "$(Write-Output hello)" (failed in the 2.3.0-shuv.1 publish, run 43492639481). This is a pwsh call that sometimes stalls for more than 15s on the runner. It is not a regression and not cold-start latency. The cause is not yet identified, so the hang detector now reports what the call was waiting on.

Stopped-command race

To make the race deterministic, I temporarily put a 100ms gap between the two printfs. With the old wait (cancel after 1 byte), both variants of the test (spawned and daemon) fail exactly as on #490: Expected - 1 / Received + 0, with the second line missing. With the new wait, both pass. On the real command, bun run test test/tool-shell.test.ts -t "ShellTool stopped" --rerun-each 30 passed 570 of 570.

PowerShell stall

  • The code path is unchanged. The failing test, runPermissionCommand, the permission and shell services, ShellSelect, the scanner and packages/util are identical between the last passing publish (ac4077949d) and 37f5865e0c.
  • It happens again, always on the same test. It failed a second time in run 43509382846 on this PR's first commit, again on Write-Output "$(Write-Output hello)". That is the file's first PowerShell test, and so the first pwsh launch of the core test process. That report showed no pending permission, a pwsh process spawned at once (15,020ms elapsed), and zero bytes of output. On the same image it has passed in 8 other full unit runs: 43497096412, 43498490451, 43501674041, 43505609084 (which also ran 660 in-situ PowerShell reruns, with no failures and a slowest run of 4.25s), 43510855667, 43513808639, 43513820154 and 43513831524. That makes 2 failures in 10 runs.
  • It is not cold start. Fresh-HOME pwsh launches during the unit suite on the runner took 1.34s for the first launch on the VM, 1.0 to 1.9s after dropping the page cache, and 0.6 to 1.5s warm, with a worst case of 5.8s. The failing test itself took 1.14s cold and 0.80s warm. Raising the timeout would turn a 15s hang into a longer one.
  • Ruled out: machine-wide load (the neighbouring turbo tasks ran at normal speed), pwsh telemetry (a blackholed proxy has no effect), blocking syslog writes, Bun losing child exits in normal operation (worst gap 427ms over 150 pwsh spawns), and a saved-approval race. Upstream's check-test unit (linux) passed at v2.3.0 (43397035564).

The first commit's report used kill(pid, 0), which also succeeds for a zombie. So it could not tell "pwsh hung before writing" apart from "pwsh exited but Bun never observed the exit or output". The report on this head distinguishes them:

  • pending: permission requests still waiting.
  • shells: the shell created and exit events seen during the call, with elapsed milliseconds.
  • commands: for each created command, its shell and job status, its pid, the process state from ps -o stat=,wchan=, and the last 500 bytes of its output file.
Report Where the call stalled
A pending request, no shell events In the permission flow
No pending request, no shell events Before spawning
Command and job both running, process S/R pwsh itself is stuck (wchan names where)
Command and job both running, process Z or gone The process ended, but Bun never delivered its exit or EOF
Command exited, job still running In settling the job

I checked the report by injecting a stall of each kind. An unapproved command gave {"pending":[["Get-Date","Write-Output hello"]],"shells":[],"commands":[]}. A 20s pwsh sleep gave created at 22ms, status and job both running, process Ssl futex_wait, and tail "starting\n".

No feature flag: this changes tests only.

Tests

  • bun run test test/tool-shell.test.ts in packages/core passes 110 of 110, and ShellTool stopped with --rerun-each 30 passes 570 of 570.
  • bun typecheck, Prettier and git diff --check pass. The pre-push hook ran the full bun run check: 40 of 40 typecheck tasks, 0 lint errors.
  • PR checks don't run unit tests (test.yml runs only for releases), so I dispatched the unit workflow by hand:
  • Locally, the exact CI unit command with --continue passed 30 of 32 tasks. The two failures exist only on this machine (GitLab OAuth tests need port 8080, and an sdk Workerd timing test); CI passes both.
  • An Opus review of the PowerShell report found nothing blocking. Its suggestions are all applied: ps process state, shell and job state, logging the report, the real output tail, a shared timeout constant, and a wider comment.

shuv1337 and others added 4 commits November 13, 2026 13:16
Co-authored-by: Cursor <cursoragent@cursor.com>
…alled permission test

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
@shuv1337 shuv1337 added the full-tests Run the full test suite (unit and e2e) on this PR label Nov 13, 2026
@shuv1337 shuv1337 changed the title test(core): report what a stalled shell permission test was waiting on test(core): fix the stopped-command race and report stalled PowerShell calls Nov 13, 2026
@shuv1337
shuv1337 merged commit f22976b into integration-v2 Nov 13, 2026
13 of 15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

full-tests Run the full test suite (unit and e2e) on this PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant