[Bug]: Every call to a linked environment waits out a silent first address (10 s) before using the address that works #35519

Before submitting

• I searched existing issues and did not find a duplicate.
• I included enough detail to reproduce or investigate the problem.

Area

apps/server

Steps to reproduce

1. Save this test as apps/server/src/peer/peer-resolve-repro.test.ts. It links to a test peer at two loopback addresses, makes the first one stop answering, and resolves the link twice on the test clock:

import { NodeHttpServer } from "@effect/platform-node";
import { expect, it } from "@effect/vitest";
import * as Context from "effect/Context";
import * as Duration from "effect/Duration";
import * as Effect from "effect/Effect";
import * as Fiber from "effect/Fiber";
import * as Layer from "effect/Layer";
import * as Queue from "effect/Queue";
import * as Ref from "effect/Ref";
import * as TestClock from "effect/testing/TestClock";
import {
  FetchHttpClient,
  HttpClient,
  HttpServer,
  HttpServerRequest,
  HttpServerResponse,
} from "effect/http";

import * as PeerLinks from "./PeerLinks.ts";
import { descriptorOf, layerLinkingEnvironment, servePeer } from "./PeerLinks.testkit.ts";

const laptop = descriptorOf("environment-laptop", "Laptop");
const box = descriptorOf("environment-box", "Box");

/** A second address of the same peer, on a socket of its own. */
const serveSecondAddress = (body: string) =>
  Layer.build(Layer.fresh(NodeHttpServer.layerTest)).pipe(
    Effect.flatMap((server) =>
      Effect.gen(function* () {
        yield* HttpServer.serve(
          Effect.as(
            HttpServerRequest.HttpServerRequest,
            HttpServerResponse.text(body, { contentType: "application/json" }),
          ),
        ).pipe(Layer.build);
        const address = (yield* HttpServer.HttpServer).address;
        if (address._tag === "UnixPathAddress") return yield* Effect.die("expected a TCP address");
        return `http://127.0.0.1:${address.port}`;
      }).pipe(Effect.provideContext(server)),
    ),
  );

it.effect("every resolve waits out a silent first address before trying the live second", () =>
  Effect.scoped(
    Effect.gen(function* () {
      const b = yield* servePeer(box, Layer.empty);
      const described = yield* HttpClient.get(`${b.url}/.well-known/t3/environment`).pipe(
        Effect.flatMap((response) => response.text),
        Effect.provide(FetchHttpClient.layer),
      );
      const second = yield* serveSecondAddress(described);
      const links = Context.get(
        yield* Layer.build(layerLinkingEnvironment(laptop)),
        PeerLinks.PeerLinks,
      );
      const pairing = yield* b.auth.issuePairingCredential();
      const linked = yield* links.link({
        url: b.url,
        alternateUrls: [second],
        pairingCode: pairing.credential,
        access: "auto",
      });
      expect(linked.urls).toEqual([b.url, second]);

      // The first address stops answering, as a tailnet address does while that route is down.
      yield* Ref.set(b.probes, 0);
      yield* Ref.set(b.asleep, true);
      console.log("list:", (yield* links.list).map((link) => link.status));
      yield* Queue.clear(b.unanswered);

      // Loopback addresses get a 3 s probe timeout; any other address gets 10 s.
      for (const call of [1, 2]) {
        const resolving = yield* links.resolve(box.environmentId).pipe(Effect.forkScoped);
        yield* Queue.take(b.unanswered);
        yield* TestClock.adjust(Duration.millis(2_900));
        console.log(`resolve #${call} at 2.9 s:`, resolving.pollUnsafe()?._tag ?? "still waiting");
        yield* TestClock.adjust(Duration.millis(100));
        const resolved = yield* Fiber.join(resolving);
        console.log(`resolve #${call} at 3.0 s:`, resolved.url === second ? "second address" : "?");
        expect(resolved.url).toBe(second);
      }
      console.log("descriptor requests the silent first address received:", yield* Ref.get(b.probes));
    }),
  ).pipe(Effect.provide(NodeHttpServer.layerTest)),
);

2. From the repository root, run node_modules/.bin/vp test run apps/server/src/peer/peer-resolve-repro.test.ts --silent=false.
3. The test passes, which confirms the behavior. Its log shows that listing reports the link reachable at once, while both resolves wait the full probe timeout on the silent first address before returning the second one.

Deterministic: same output on each of 3 runs. The peers are the repository's own PeerLinks.testkit.ts test servers on loopback, so the wait is the 3-second loopback timeout. A peer at a tailnet or other non-loopback address waits 10 seconds per call (PeerLinks.ts:53-54, source-only for that value).

Expected behavior

Once a linked environment's later address answers as the peer, calls to it should not each wait the full probe timeout of an earlier address that has stopped answering. A forwarded call should reach the working address about as fast as listing finds it, since listing already races the addresses (PeerLinks.ts:574-588, the fix for #28531). This mirrors the client's own route handling, where a route that does not answer is "checked briefly and skipped" (docs/user/remote-access.md, "Reach one machine several ways"). The token must still go only to an address that answers as this peer (PeerLinks.ts:644-645). The exact bound on the delay is an unverified expectation.

Actual behavior

A peer link stores every address that answered as the peer when it was linked, for example the peer's Tailscale HTTPS name https://host-a.tailnet-name.ts.net followed by its T3 Connect address. When the first stored address stops answering, as a tailnet route does when that path goes down, every forwarded MCP call waits for that address's probe to time out before it tries the next one, and the next call does the same again. PeerLinks.resolve checks the stored addresses one at a time with Effect.findFirst (apps/server/src/peer/PeerLinks.ts:644-657). Each check is a descriptor request with one retry, bounded by probeTimeout, which is 10 seconds for any non-loopback address and 3 seconds for loopback (apps/server/src/peer/PeerLinks.ts:53-54, :225-247). Nothing records which address answered, so the next resolve starts at the silent address again. PeerMcpClient.call runs resolve for every call (apps/server/src/peer/PeerMcpClient.ts:163-166), and a forwarded t3_thread_wait is sent in pieces of at most 50 seconds, each its own call (apps/server/src/peer/PeerForwarding.ts:33, :199-206). Listing races the same addresses with a 3-second cap (apps/server/src/peer/PeerLinks.ts:574-588), so Settings shows the link as Reachable while each agent call to it stalls.

Impact and scope: Impact: P2 - every agent call to a linked environment is delayed by the probe timeout of its silent first address, 10 seconds for a tailnet or other remote address, until that address answers again; a workaround exists.

Every forwarded tool call (t3_thread_read, launch, send, interrupt, and others through PeerForwarding) pays the delay. A forwarded t3_thread_wait pays it once per 50-second piece, so a 10-minute wait spends about 2 extra minutes on resolution. This happens whenever the first stored address is silent, for example a Tailscale address while the tailnet path between the two machines is down and the T3 Connect address still works. Workaround: forget the link and link again by typing only the working address, which drops the tailnet route from the link.

Impact

Minor bug or occasional failure

Version or commit

main @ 86ff9d0

Environment

Reproduced with the repository's Vitest setup (vp test, Vite+ 5.0.5) and @effect/platform-node loopback test servers on Linux x64. The code path has no platform-specific branches.

Logs or stack traces

Test log from step 2:

list: [ 'reachable' ]
resolve #1 at 2.9 s: still waiting
resolve #1 at 3.0 s: second address
resolve #2 at 2.9 s: still waiting
resolve #2 at 3.0 s: second address
descriptor requests the silent first address received: 3
      Tests  1 passed (1)

The silent address received one request from listing and one from each resolve; the second resolve did not skip it.

Order of stored addresses (source reading): the Link dialog sends the machine's addresses in the order the client lists them: the server's advertised endpoints first, then saved routes, then the T3 Connect address last (packages/client-runtime/src/linkRequest.ts:120-141). PeerLinks.link keeps those that answered, in that order (apps/server/src/peer/PeerLinks.ts:409-453, :515-519). A tailnet address therefore comes before the T3 Connect address.

Screenshots, recordings, or supporting files

No response

Workaround

See Impact and scope under Actual behavior.

Additional details

• Interface: MCP: tools forwarded to a linked environment (Linked environments, Settings → Labs).
• Earliest failure stage: Integration or transport

References

#26551 (open PR, unmerged at the commit below, that proposes reusing a verified fallback route for 60 seconds; no issue tracks this behavior), #28531 (closed; the same wait in listing, fixed by racing the addresses, which resolve does not do), #26450 and #30870 (open; linking refuses a Tailscale address, a different failure at link time)

Found by reading T3 Code's Tailscale integration against the Tailscale source (tailscale/tailscale@071bf947, v1.105.0); written up with AI assistance (Claude Code).