Add a JSON-RPC server mode to F# Interactive

[xperiandri]

Description

Adds a JSON-RPC server mode to F# Interactive: --fsi-server-jsonrpc:<pipe name>.

This is the first stage of replacing the Visual Studio F# Interactive window. It changes only
F# Interactive, is useful to any editor on its own, and carries no dependency on the window work
that will build on it. That work — a window hosted on Microsoft.VisualStudio.InteractiveWindow,
the same REPL engine C# Interactive and Python Interactive run on — will come as a separate PR.

Why

An editor hosting F# Interactive today drives it through standard input and recovers results by
looking for a SERVER-PROMPT> marker in the output text. That protocol cannot say which output
belongs to which submission, carries no structured diagnostics, and has pushed the Visual Studio
window into a set of workarounds that are worth listing, because each one disappears here:

• a line directive wrapped around every selection sent from the editor;
• a temporary file that the first submission writes its own process id into, because under
  dotnet fsi the process the IDE launched is not the process evaluating code — and the IDE then
  polls for that file in a Thread.Sleep loop;
• counting output lines to skip the val it: unit = () that the process-id discovery itself
  produced;
• a separate remoting channel (CtrlBreakClient) carrying interrupts alongside the text protocol.

What the mode does

The host submits interactions over a named pipe and receives results as data. Program output keeps
flowing through the redirected console streams, so the control channel and the script's own output
no longer have to be told apart after the fact. The prompt is suppressed in this mode, so nothing
has to be filtered back out of the output either.

Method	Result
fsi/initialize	the evaluating process id, framework description, architecture, fsi version, working directory
fsi/execute	success, cancelled, diagnostics, escaping exception, working directory
fsi/executeFile	as above
fsi/setPaths	as above
fsi/interrupt	whether an interaction was interrupted
fsi/shutdown	—

Three details are worth calling out:

• processId is the process evaluating code. Under dotnet fsi that is not the process the
  host launched, and it is what a debugger has to attach to. Reporting it in the handshake is what
  removes the temporary-file dance.
• sourcePath and startLine make the session emit the line directive itself, so a selection
  executed from an editor reports its errors against the user's own file and line.
• exception is omitted when the interaction merely failed to compile. The diagnostics already
  describe that, and a host reporting both would say the same thing twice.

Transport

StreamJsonRpc over a HeaderDelimitedMessageHandler, the same combination Roslyn's interactive host
uses. The protocol types live in one file compiled into fsi, so a host can share the source rather
than restate them. The version is pinned to the one Roslyn's packages already force into
vsintegration, so the two agree.

This adds eight assemblies (~3 MB) next to fsi.dll: StreamJsonRpc, Newtonsoft.Json, MessagePack
and its annotations, Nerdbank.MessagePack, Nerdbank.Streams, Microsoft.VisualStudio.Threading and
Microsoft.VisualStudio.Validation. Worth naming explicitly, since fsi ships inside the SDK. The
alternative was framing and dispatch maintained here, against a library that is already a dependency
of the editor on the other end of the same connection.

Threading

Unchanged from a console session in the way that matters: interactions are evaluated on the event
loop thread through EventLoopInvoke, exactly as the standard input path already does, so scripts
that create user interface objects behave as they do at the console.

Interactions queue onto a single worker so that they run in the order they arrived, and complete
their tasks there. That leaves StreamJsonRpc free to dispatch fsi/interrupt while an interaction
is still running — an interrupt that waited its turn behind the interaction it is meant to stop
would never arrive.

Changes to src/Compiler/Interactive/fsi.fs

Five small hunks, all guarded by the new option, no public API change:

• register --fsi-server-jsonrpc;
• treat the new mode as a server mode for the purpose of not using the console reader;
• suppress the prompt, since the host learns an interaction finished from the response to its
  request;
• skip the standard input reader thread, leaving standard input to the script being run;
• keep the legacy interrupt channel for the legacy server mode only.

Lifetime

The session watches the process id it was given and exits when it goes, so a crashed editor does not
leave an orphan behind. Closing the control channel has the same effect.

Tests

tests/FSharp.Compiler.Interactive.Server.Tests — 23 tests that launch a real fsi process and drive
the protocol over the pipe. The parts most likely to break are the ones that only exist across a
process boundary, so an in-process stand-in would not have exercised them: the handshake, the
lifetime of the session, and the interaction between the control channel and the output streams.

Covered: state kept across interactions, structured diagnostics with error numbers and positions,
diagnostics attributed to the host's own file and line, escaping exceptions, warnings not failing an
interaction, text that has to survive JSON escaping, #load, working-directory changes, interrupting
a running interaction, refusing requests before the handshake, refusing unknown methods, and exiting
when the host disconnects or its process dies.

Checklist

• Test cases added
• Performance benchmarks added in case of performance changes
• Release notes entry updated

[github-actions]

❗ Release notes required

You can open this PR in browser to add release notes: open in github.dev

---

✅ Found changes and release notes in following paths:

Change path	Release notes path	Description
`src/Compiler`	docs/release-notes/.FSharp.Compiler.Service/11.0.100.md

[majocha]

An idea: we could initially make it an out of process part of VS, a fsi session wrapper alternative to fsmain.

[github-actions]

🔍 Tooling Safety Check — Affects-Build-Infra, Affects-Restore, Affects-Test-Tooling
Affects-Build-Infra: adds PackageReference in fsi.targets, modifies FSharp.slnx
Affects-Restore: new package version in eng/Packages.props
Affects-Test-Tooling: adds new test project to solution

Generated by PR Tooling Safety Check · opus46 5.3M · ◷