diff --git a/CHANGELOG.md b/CHANGELOG.md index 499ed8da0a..85d2d9450a 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -41,6 +41,18 @@ file is the human-readable summary of what changed and why. ### Provider setup +- **Images reach the gateway instead of being dropped** — the provider block + written by the netGate configure flow and the custom-provider flow now + declares `attachment: true` and `modalities: { input: ["text", "image"] }` + for every model it lists. A config model entry with no modalities resolves to + text-only, so an attached image used to be replaced with an "ERROR: Cannot + read image" note before the request and the model would answer that it can't + see anything. Every gateway model is now declared vision-capable, including + text-only ones: for those the endpoint rejects or ignores the image rather + than the client silently dropping it. Configs written before this change need + one more pass through the configure flow (or a CoDev Hub config rewrite, + which does the same since hub PR #219) to pick the declaration up. + - **The gateway provider is now netGate** — the SSO configure flow (and its manual key-paste branch) writes the provider block as `provider.netgate` with display name `netGate` instead of `provider.aigateway` / `AI Gateway`, diff --git a/DIVERGENCES.md b/DIVERGENCES.md index 91fa67e441..834d8da844 100644 --- a/DIVERGENCES.md +++ b/DIVERGENCES.md @@ -155,6 +155,7 @@ from the squash message first, or use a plain merge. | `packages/core/test/ripgrep.test.ts` | `120_000` timeout on both `it.live` tests | First use downloads + extracts the ripgrep binary; GitHub-hosted Windows runners blow bun's 5s default. | Keep our timeouts; take upstream's test changes. | | `packages/opencode/test/cli/run/run-process.test.ts`, `packages/opencode/test/cli/acp/lifecycle.test.ts`, `packages/opencode/test/server/httpapi-file.test.ts`, `packages/opencode/test/server/httpapi-sdk.test.ts`, `packages/opencode/test/effect/runner.test.ts`, `packages/opencode/test/session/prompt.test.ts`, `packages/opencode/test/project/instance-bootstrap.test.ts` | Slow-runner headroom: unknown-model test `timeoutMs`/bound 15s → 30s (test timeout 60s); the two unknown-stream-finish `codev run` subprocess tests ("unknown stream finish preserves partial output…", "--format json records partial output for an unknown stream finish") `timeoutMs` 30s → 60s and test timeout 60s → 120s (pure-upstream spawn tests that pay `codev run` cold-start; neither asserts on duration, so headroom only); the three tool-executing `codev run` subprocess tests ("prints each completed text part in order around a tool continuation", "--format json preserves reasoning, tool, and continuation ordering", "rejects requested permissions by default and allows them with the dangerous flag") `timeoutMs` 30s (harness default) → 60s explicitly, test timeout 60s → 120s — 240s for the permissions test, which pays three sequential `codev run` spawns; these pay cold-start *plus* a real `bash` tool spawn and a continuation turn, and were timing out at exactly the 30s harness default on loaded runners; none asserts on duration, so headroom only; ACP stdin-EOF exit timeout 5s → 30s; file-search-index poll 5s → 120s with a 150s test timeout (the search-endpoints test pays ripgrep's first-use download on Windows runners, like ripgrep.test.ts); Runner fiber-coordination `Effect.timeout` guards 250ms/1s → 10s (pure hang-detectors on already-unblocked awaits; Windows runners starve the event loop past 250ms); prompt.test.ts's two `sleep`-based loop/shell tests ("loop waits while shell runs…", "shell completion resumes queued loop callers") moved from `it.instance` to the file's own `unix` helper so they skip on Windows like their `unixNoLLMServer` siblings (`sleep` isn't on Windows, so the shell fiber hangs to the budget — a genuine hang, not slowness); instance-bootstrap's first test 30s → 60s (pays instance cold-start) | Loaded GitHub-hosted runners spawn processes, fetch ripgrep, and build the lazy file index slower than upstream's Blacksmith runners; 250ms–30s budgets flake or fail outright, and Unix-shell tests can't run on Windows at all. Hang regressions are still caught (a hang exceeds the bound; polls return as soon as ready). | Keep our numbers and the `unix` gating; take upstream's test changes. | | `packages/opencode/test/tool/external-directory.test.ts`, `packages/opencode/test/tool/read.test.ts` | Windows path-variant tests keep the drive letter in the `alt` form (drop the `.replace(/^[A-Za-z]:/, "")` step) | A drive-less path is drive-relative: it resolves against the CWD's drive (D: on GitHub-hosted Windows) while the temp dir lives on C:, so the file genuinely isn't there. Slash + case variants still exercise normalization. | Keep our `alt` construction; take upstream's test changes. | +| `packages/opencode/test/provider/provider.test.ts`, `packages/opencode/test/provider/transform.test.ts` | Appended two fork-only blocks at the end of each file: an `it.instance` ("declared image modality makes a custom-provider model attachment capable") asserting a config model with no `modalities` resolves `attachment`/`input.image` to false while one carrying the `@codev/gateway` block resolves both true, and a `describe` ("ProviderTransform.message - image input on a custom provider") asserting `unsupportedParts` passes `image`/`file` parts through when the model declares image input and swaps them for the `ERROR: Cannot read …` note when it doesn't. No upstream test changed. | These pin the two upstream code paths the fork's config writer depends on (`provider.ts` modality→capability resolution, `transform.ts` part filtering); upstream leaves both untested, so a refactor there would silently re-break gateway image attachments. | Keep our appended blocks; take upstream's test changes. Append-only, so conflicts should be end-of-file context alone. | | `packages/opencode/test/server/httpapi-exercise/environment.ts` | Exercise config/data dirs `config/opencode`,`data/opencode` → `…/codev`; seeds `codev.json` with `{"share":"manual"}` | Follows the `global.ts` app-dir rename (row above); the fork's `share: "disabled"` default (config.ts row) would 500 the share/unshare scenarios, so the isolated environment opts back in (network stays stubbed by `CODEV_DISABLE_SHARE`). Seed file follows the config-file rename row (global `config.json` is no longer read). | Keep our dir names + seeded config; take upstream's surrounding changes. | | `sdks/vscode/package.json`, `sdks/vscode/src/extension.ts`, `sdks/vscode/README.md` | `name` `opencode` → `codev-code` (the extension ID is `publisher.name`); `displayName` "opencode" → "CoDev Code"; `publisher` `sst-dev` → `CoDev` (PLACEHOLDER — no such Marketplace publisher is registered; the mixed-case `CoDev` also isn't a valid Marketplace publisher id, which must be lowercase — fine for the sideloaded `.vsix` the fork ships, but replace with the real fork-owned lowercase publisher id before any `vsce publish`, and keep it in sync with the CLI auto-install target); `description` "opencode for VS Code" → "CoDev Code for VS Code"; `icon` `images/icon.png` is a fork-local generated PNG of the CoDev "c" badge (brand teal `#0891B2` on `#0a0a0a`, matching the `codev --mini` splash), with `images/icon.svg` as its source of record — replacing the earlier symlink to the shared `packages/identity` bracket mark; the two per-theme command `icon` objects (`{light,dark}` → `images/button-{dark,light}.svg`) collapse to a single `"images/button.svg"` string and `extension.ts`'s terminal `iconPath` object collapses to a single `Uri`, both pointing at a fork-local transparent teal `#0891B2` "c" glyph that reads on light+dark themes (replacing the two `packages/identity` bracket-mark symlinks, now deleted); `repository.url` → `github.com/quickbeard/codev-code`; command IDs `opencode.*` → `codev.*` across `contributes.commands`/`menus`/`keybindings` and the `registerCommand` calls; command/keybinding titles "Open opencode…"/"Run opencode"/"opencode: Insert At-Mentioned" → "Open CoDev Code…"/"Run CoDev Code"/"CoDev Code: Insert At-Mentioned" (keys unchanged); `TERMINAL_NAME` `opencode` → `codev` and the terminal launches `codev --port `; scripts gain `"vsix"` (`vsce package --no-dependencies --skip-license -o dist/`) with `@vscode/vsce` added as a pinned devDependency; README body rebranded and trimmed to the details-panel essentials — CoDev Code intro linking the fork repo, `npm install -g codev-code` prerequisite, and feature prose only; upstream's Support and Development/Making-Changes sections are dropped (they're contributor-facing, not end-user extension description) | The fork ships this extension as an installable `.vsix`, so every user-visible surface — palette titles, command IDs, terminal name, launched binary, details-panel README — must say codev, and the launched binary must be the fork's `codev` CLI. Distinct `codev.*` IDs also avoid clashing with upstream's extension if both are installed. `--no-dependencies` because the bundle is self-contained (no runtime deps, bun-managed lockfile); `--skip-license` because `sdks/vscode` has no LICENSE file and vsce aborts without it. | Keep our name + displayName + publisher + repository + description, command IDs + titles, `TERMINAL_NAME`, the `codev --port` launch line, the `vsix` script + `@vscode/vsce` devDependency, and the trimmed README (branding + dropped Support/Development sections); take upstream's extension-logic and manifest-structure changes. | | `packages/opencode/src/ide/index.ts` | IDE auto-install target `sst-dev.opencode` → `CoDev.codev-code` (`publisher.name` from `sdks/vscode/package.json`) | The CLI must install the fork's extension ID (`publisher.name` from `sdks/vscode/package.json`), not upstream's Marketplace listing. `Ide.install` currently has no runtime caller (only its test imports the module), and the install only resolves once the extension is actually published — until then the local `.vsix` is the supported install path. | Keep our extension ID (and re-sync it if the registered publisher id changes); take upstream's install/detection logic changes. | diff --git a/packages/codev-gateway/src/config-block.ts b/packages/codev-gateway/src/config-block.ts index c183a2f8f6..a744696cc2 100644 --- a/packages/codev-gateway/src/config-block.ts +++ b/packages/codev-gateway/src/config-block.ts @@ -1,6 +1,23 @@ import { GATEWAY_COMPACT_RESERVED, GATEWAY_CONTEXT_WINDOW, GATEWAY_MAX_OUTPUT_TOKENS } from "./const" import { GATEWAY_PROVIDER } from "./provider" +// A config model entry with no `modalities` resolves image input to false, and +// the session then swaps an attached image for an "ERROR: Cannot read image" +// note before the request instead of sending it. Declare image input so +// attachments reach the endpoint; a text-only model there rejects or ignores +// them, which beats the client silently dropping them. `attachment` is what +// lets the TUI attach files in the first place. The annotation keeps the arrays +// from widening to string[], which the config schema's modality literals reject. +type Modality = "text" | "image" + +const MODEL_MODALITIES: { attachment: boolean; modalities: { input: Modality[]; output: Modality[] } } = { + attachment: true, + modalities: { + input: ["text", "image"], + output: ["text"], + }, +} + // The exact codev.json fragment the codev hub writes for CoDev Code (hub // src/lib/configure.ts configureOpenCodeKind), so hub-managed and // self-configured installs are indistinguishable — the hub marks a config as @@ -8,7 +25,8 @@ import { GATEWAY_PROVIDER } from "./provider" // id, `netgate`, `ai-gateway`, legacy `aigateway`; hub PR #212), so hubs older // than that PR cannot attribute a `netgate` config. The per-model `limit` is // mandatory: without it opencode sizes the context window to 0 and disables -// auto-compaction. +// auto-compaction; the modality block is mandatory for the same reason (hub PR +// #219) — see MODEL_MODALITIES. export function buildGatewayConfig(input: { apiKey: string; baseUrl: string; model: string; models?: string[] }) { const ids = [...new Set([input.model, ...(input.models ?? [])])] return { @@ -30,6 +48,7 @@ export function buildGatewayConfig(input: { apiKey: string; baseUrl: string; mod id, { name: id, + ...MODEL_MODALITIES, limit: { context: GATEWAY_CONTEXT_WINDOW, output: GATEWAY_MAX_OUTPUT_TOKENS, @@ -48,7 +67,8 @@ export function buildGatewayConfig(input: { apiKey: string; baseUrl: string; mod // in by provider id at load) and no npm (the loader defaults to // @ai-sdk/openai-compatible). The per-model limit is still mandatory — without // it opencode sizes the context window to 0 — so the gateway defaults stand in -// for the unknown endpoint's real limits. +// for the unknown endpoint's real limits, and MODEL_MODALITIES applies for the +// same reason it does to the gateway block. export function buildCustomProviderConfig(input: { providerID: string; name: string; baseUrl: string; models: string[] }) { return { provider: { @@ -62,6 +82,7 @@ export function buildCustomProviderConfig(input: { providerID: string; name: str id, { name: id, + ...MODEL_MODALITIES, limit: { context: GATEWAY_CONTEXT_WINDOW, output: GATEWAY_MAX_OUTPUT_TOKENS, diff --git a/packages/codev-gateway/test/config-block.test.ts b/packages/codev-gateway/test/config-block.test.ts index 59423a4b22..ee4447bd0a 100644 --- a/packages/codev-gateway/test/config-block.test.ts +++ b/packages/codev-gateway/test/config-block.test.ts @@ -30,14 +30,35 @@ describe("buildGatewayConfig", () => { name: "netGate", options: { baseURL: "https://gw.test/v1", apiKey: "sk-test" }, models: { - "MiniMax/MiniMax-M2.7": { name: "MiniMax/MiniMax-M2.7", limit: { context: 196608, output: 65536 } }, - "other/model": { name: "other/model", limit: { context: 196608, output: 65536 } }, + "MiniMax/MiniMax-M2.7": { + name: "MiniMax/MiniMax-M2.7", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, + "other/model": { + name: "other/model", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, }, }, }, }) }) + // Without the declaration a custom-provider model resolves to text-only and + // the session replaces an attached image with an error note (hub PR #219). + test("declares image input so attachments are not stripped", () => { + const models = buildGatewayConfig({ apiKey: "k", baseUrl: "https://gw.test", model: "m", models: ["m", "n"] }) + .provider.netgate.models + for (const id of ["m", "n"]) { + expect(models[id].attachment).toBe(true) + expect(models[id].modalities).toEqual({ input: ["text", "image"], output: ["text"] }) + } + }) + test("falls back to a single-model map and dedupes the default", () => { const config = buildGatewayConfig({ apiKey: "k", baseUrl: "https://gw.test/v1", model: "m" }) expect(Object.keys(config.provider.netgate.models)).toEqual(["m"]) @@ -60,8 +81,18 @@ describe("buildCustomProviderConfig", () => { name: "My Gateway", options: { baseURL: "https://gw.test/v1" }, models: { - m1: { name: "m1", limit: { context: 196608, output: 65536 } }, - m2: { name: "m2", limit: { context: 196608, output: 65536 } }, + m1: { + name: "m1", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, + m2: { + name: "m2", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, }, }, }, diff --git a/packages/opencode/test/provider/provider.test.ts b/packages/opencode/test/provider/provider.test.ts index d5af652a17..d290982ada 100644 --- a/packages/opencode/test/provider/provider.test.ts +++ b/packages/opencode/test/provider/provider.test.ts @@ -668,6 +668,47 @@ it.instance( }, ) +// The netGate and custom-provider config blocks written by @codev/gateway +// (config-block MODEL_MODALITIES) exist to flip these two flags: without the +// declaration a config model is text-only and ProviderTransform.message drops +// attached images before the request. +it.instance( + "declared image modality makes a custom-provider model attachment capable", + Effect.gen(function* () { + const providers = yield* list + const models = providers[ProviderV2.ID.make("netgate")].models + const bare = models["bare-model"] + expect(bare.capabilities.attachment).toBe(false) + expect(bare.capabilities.input.image).toBe(false) + const declared = models["declared-model"] + expect(declared.capabilities.attachment).toBe(true) + expect(declared.capabilities.input.image).toBe(true) + expect(declared.capabilities.input.text).toBe(true) + expect(declared.capabilities.output.image).toBe(false) + }), + { + config: { + provider: { + netgate: { + name: "netGate", + npm: "@ai-sdk/openai-compatible", + env: [], + models: { + "bare-model": { name: "bare-model", limit: { context: 196608, output: 65536 } }, + "declared-model": { + name: "declared-model", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, + }, + options: { baseURL: "https://gw.test/v1", apiKey: "sk-test" }, + }, + }, + }, + }, +) + it.instance( "model with custom cost values", Effect.gen(function* () { diff --git a/packages/opencode/test/provider/transform.test.ts b/packages/opencode/test/provider/transform.test.ts index 764ec5dce4..e3da42009f 100644 --- a/packages/opencode/test/provider/transform.test.ts +++ b/packages/opencode/test/provider/transform.test.ts @@ -5247,3 +5247,80 @@ describe("ProviderTransform.options - kimi family adaptive thinking", () => { expect(result.thinking).toBeUndefined() }) }) + +// The behavior the netGate/custom-provider config block buys: a model entry +// with no `modalities` resolves image input to false, and the image never +// reaches the endpoint. See @codev/gateway config-block MODEL_MODALITIES. +describe("ProviderTransform.message - image input on a custom provider", () => { + const createModel = (image: boolean) => + ({ + id: "MiniMax/MiniMax-M2.7", + providerID: "netgate", + api: { + id: "MiniMax/MiniMax-M2.7", + url: "https://gw.test/v1", + npm: "@ai-sdk/openai-compatible", + }, + name: "MiniMax/MiniMax-M2.7", + capabilities: { + input: { text: true, audio: false, image, video: false, pdf: false }, + output: { text: true, audio: false, image: false, video: false, pdf: false }, + }, + options: {}, + headers: {}, + }) as any + + const pngDataUrl = "data:image/png;base64,iVBORw0KGgo=" + + test("passes an image part through when the model declares image input", () => { + const result = ProviderTransform.message( + [{ role: "user", content: [{ type: "text", text: "what is this" }, { type: "image", image: pngDataUrl }] }] as any[], + createModel(true), + {}, + ) as any[] + expect(result[0].content).toEqual([ + { type: "text", text: "what is this" }, + { type: "image", image: pngDataUrl }, + ]) + }) + + test("replaces an image part with an error note when it does not", () => { + const result = ProviderTransform.message( + [{ role: "user", content: [{ type: "image", image: pngDataUrl }] }] as any[], + createModel(false), + {}, + ) as any[] + expect(result[0].content).toEqual([ + { type: "text", text: "ERROR: Cannot read image (this model does not support image input). Inform the user." }, + ]) + }) + + test("passes an attached image file through when the model declares image input", () => { + const part = { type: "file", mediaType: "image/png", filename: "shot.png", data: pngDataUrl } + const result = ProviderTransform.message( + [{ role: "user", content: [part] }] as any[], + createModel(true), + {}, + ) as any[] + expect(result[0].content).toEqual([part]) + }) + + test("names the dropped file in the error note when it does not", () => { + const result = ProviderTransform.message( + [ + { + role: "user", + content: [{ type: "file", mediaType: "image/png", filename: "shot.png", data: pngDataUrl }], + }, + ] as any[], + createModel(false), + {}, + ) as any[] + expect(result[0].content).toEqual([ + { + type: "text", + text: 'ERROR: Cannot read "shot.png" (this model does not support image input). Inform the user.', + }, + ]) + }) +}) diff --git a/packages/tui/test/custom-provider.view.test.tsx b/packages/tui/test/custom-provider.view.test.tsx index 32075d5105..1f084426d1 100644 --- a/packages/tui/test/custom-provider.view.test.tsx +++ b/packages/tui/test/custom-provider.view.test.tsx @@ -165,7 +165,7 @@ test("Custom provider with a base URL discovers models and writes the codev.json boot_.setup.mockInput.pressArrow("down") // Custom provider boot_.setup.mockInput.pressEnter() - await waitFrame(boot_.setup, (f) => f.includes("Provider id")) + await waitFrame(boot_.setup, (f) => f.includes("Provider name")) await answer(boot_.setup, "mockgw") await waitFrame(boot_.setup, (f) => f.includes("API key")) await answer(boot_.setup, "sk-mock") @@ -174,16 +174,29 @@ test("Custom provider with a base URL discovers models and writes the codev.json await waitUntil(boot_.setup, () => boot_.captured.config.length > 0) - // The credential goes to auth.json, keyless; the provider block goes to codev.json. + // The credential goes to auth.json, keyless; the provider block goes to + // codev.json — with the image declaration, without which the discovered + // models would be text-only and attachments dropped before the request. expect(boot_.captured.auth).toEqual([{ providerID: "mockgw", body: { type: "api", key: "sk-mock" } }]) expect(boot_.captured.config).toHaveLength(1) expect(boot_.captured.config[0]).toEqual({ provider: { mockgw: { + name: "mockgw", options: { baseURL: `${modelsBase}/v1` }, models: { - "mock-small": { name: "mock-small", limit: { context: 196608, output: 65536 } }, - "mock-large": { name: "mock-large", limit: { context: 196608, output: 65536 } }, + "mock-small": { + name: "mock-small", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, + "mock-large": { + name: "mock-large", + attachment: true, + modalities: { input: ["text", "image"], output: ["text"] }, + limit: { context: 196608, output: 65536 }, + }, }, }, }, @@ -206,7 +219,7 @@ test("Custom provider with no base URL stores only the credential", async () => boot_.setup.mockInput.pressArrow("down") // Custom provider boot_.setup.mockInput.pressEnter() - await waitFrame(boot_.setup, (f) => f.includes("Provider id")) + await waitFrame(boot_.setup, (f) => f.includes("Provider name")) await answer(boot_.setup, "credx") await waitFrame(boot_.setup, (f) => f.includes("API key")) await answer(boot_.setup, "sk-cred")