feat(ee): LAN compute cluster

[ganisback]

Closes #172.

Combines the compute boxes on a LAN into one pool, so a request that arrives
at any machine is served by whichever member can answer it soonest. Their IP
addresses change on every reboot, so nothing in the design depends on an
address: members are identified by a UUID pinned to a self-signed certificate,
found again over mDNS, and their addresses are relearned on every poll.

What it does

Forms itself. Every machine installed with the same
CSGHUB_LITE_CLUSTER_SECRET founds or joins the same cluster with no create
or join step: the cluster UUID is derived from the secret, and the secret
itself never goes on the wire. Manual pairing by join token or by an 8-digit
admission code still works for anyone who wants it.

Stays out of the way on one machine. Without that variable, and until an
operator asks for clustering, there is no listener, no multicast and no
polling goroutine. A single machine pays nothing for a feature it is not
using.

Routes on predicted completion time, not round-robin: load time plus
queue plus prompt plus decode, with penalties for thermal throttling, power
caps, CPU contention, RAM pressure, disk activity and foreign GPU use. A
conversation sticks to the node that answered it, and a node that fails is
taken out and retried with backoff.

Copies models between members over the cluster link rather than
re-downloading from the internet, verifying a SHA-256 the sender computes
while streaming. Derived files such as a converted GGUF follow only over a
link fast enough to be worth it.

Speaks the existing APIs. Chat, embeddings, Anthropic messages,
transcription and speech all route through the engine getters that were
already there rather than a second router. A request opts in with
source: "cluster", pins a machine with source: "node:<uuid>", or names
nothing and is routed only when this node cannot serve it.

Licensing

The feature and the dashboard are open to everyone. Only the node count is
gated: quota.lite.max_cluster_nodes is 2 without a licence, unlimited with
one. The cluster code lives in ee/ under the Enterprise licence; the seam
internal/server exposes to it is a single Host interface.

Measured on two machines

An M5 and an M4 on the same 5 GHz network, entry node M5. Single machine is
source: "local", two machines is balanced mode.

Text generation, Qwen3.5-2B, 128 tokens per request:

concurrency	1 machine	2 machines	speedup	TTFT 1	TTFT 2
1	4.30s	5.37s	0.80x	0.09s	0.41s
2	8.62s	5.23s	1.65x	2.29s	0.29s
4	16.99s	10.36s	1.64x	6.48s	2.66s
8	34.43s	20.72s	1.66x	15.13s	7.51s

Embedding, Qwen3-Embedding-0.6B, 32 inputs per request:

concurrency	1 machine	2 machines	speedup
1	0.67s	0.87s	0.77x
2	1.27s	1.51s	0.84x
4	2.51s	2.10s	1.19x
8	5.45s	3.01s	1.81x
16	10.41s	5.73s	1.82x
32	20.68s	11.10s	1.86x

One machine's throughput is flat whatever the concurrency, so extra requests
only queue; the second machine roughly doubles it and halves time to first
token. Below two concurrent requests clustering costs 16-23% because there is
nothing to spread and a network hop to pay, which is why local_first is the
default and balanced is opt-in.

Peer model copy measured 1.1 GB in 37 seconds. Failover was exercised by
stopping a member: cluster-sourced requests fell back to the surviving node
and a request pinned to the stopped one returned a clear 503.

Review pass

The last six commits are an architecture and duplication review of the
preceding work, with the three real defects it found:

• A routed embeddings request placed on the local node was served by a chat
  engine, because the kind of engine never travelled with the request. A
  llama-server started without --embeddings cannot answer it.
• An AI app could not be bound to the cluster at all: the provider route
  vocabulary did not know cluster or node:<uuid> and rejected them.
• Every forwarded inference request built a fresh HTTP client and paid a
  full TLS handshake, then abandoned the transport. Members now keep one
  pooled client each, which measured 0.42s to 0.05s on the second request.

The rest removes duplication: one owner for the request source vocabulary
instead of four copies of the precedence, one dispatch helper for the routed
speech engines instead of four copies of the same skeleton, one Chat
instead of two, and one local-status cache instead of two with a stale one
the invalidation could not reach. A 429 from a member is now a cooldown that
honours Retry-After rather than an immediate retry.

Known gaps

Deliberately not addressed here, each needing a shared layer across the
Apache and Enterprise packages:

• The cluster router and the provider pool router are still two
  implementations of dispatch, failover and conversation affinity.
• Manager carries 116 methods across five files.
• About 1500 lines under internal/server are cluster logic; roughly a third
  of that must stay there because it reads private server state, the rest
  belongs in ee/.
• ee/LICENSE still needs a clause permitting production use within the
  community node quota.

Testing

go test -race ./... passes with no data races, make lint passes, 83 web
tests pass. The cluster package has 39 cases and the server has 8 cluster
integration cases, plus the two-machine runs above.

Generated with AI

Co-Authored-By: csglite xzhgan@gmail.com