Skip to content

feat(gateway): declare image input for gateway and custom provider models - #139

Merged
quickbeard merged 1 commit into
mainfrom
declare-image-modality
Jul 30, 2026
Merged

quickbeard merged 1 commit into
mainfrom
declare-image-modality

Conversation

@quickbeard

Copy link
Copy Markdown
Owner

Summary

Follows codev-hub PR #219, which fixed the same defect in the hub's config writer. This repo has a mirror of that writer (packages/codev-gateway/src/config-block.ts), so the fork's own configure flows were still emitting text-only models.

A config model entry with no modalities resolves capabilities.input.image to false (provider/provider.ts:1463), and unsupportedParts then swaps the attached image for an ERROR: Cannot read image (this model does not support image input) note before the request leaves the client (provider/transform.ts:411). The model replies that it can't see any image — even when the gateway and model support vision.

Both blocks the fork writes now declare, per model:

"attachment": true,
"modalities": { "input": ["text", "image"], "output": ["text"] }
  • buildGatewayConfig — netGate SSO / key-paste configure flow. Byte-parity with the hub writer is a hard requirement here (hub-managed and self-configured installs must be indistinguishable), so this had to move regardless.
  • buildCustomProviderConfig — the TUI "Custom provider" flow. Not hub territory, but the identical failure mode.

Trade-off

Every gateway model is now declared vision-capable, including text-only ones. For those an attached image reaches the endpoint (a possible 400 or a confused reply) instead of being stripped client-side. Intentional, and the same call the hub PR made: the gateway/model decides what to do with an image it can't use, not the client.

Configs written before this change need one more pass through the configure flow, or a CoDev Hub config rewrite, to pick the declaration up. Healing existing installs at load time would mean defaulting custom-provider models to image-capable in provider.ts — an upstream-file divergence, deliberately not done here.

Tests

  • packages/codev-gateway/test/config-block.test.ts — writer output for both builders, plus the multi-model map.
  • packages/opencode/test/provider/provider.test.ts — new it.instance: a netgate model with no modalities resolves attachment/input.image false; one with the block resolves both true. Pins the config → capability path.
  • packages/opencode/test/provider/transform.test.ts — new describe: image and file/image/png parts pass through when the model declares image input, and become the ERROR: Cannot read … note when it doesn't. unsupportedParts had no coverage.
  • packages/tui/test/custom-provider.view.test.tsx — the end-to-end PATCH body now expects the declaration on every discovered model.

Two pre-existing failures repaired in that view test (verified failing on clean main): they wait on a Provider id prompt that #137 renamed to Provider name (optional), and their expected payload predates the provider name field. Same file and same assertion this PR touches, so fixed rather than left red.

Verified: bun test test/provider/ in packages/opencode → 493 pass; packages/codev-gateway → 36 pass; custom-provider.view → 4 pass, configure-gate → 7 pass; bun typecheck clean in opencode, tui, codev-gateway.

Bookkeeping

  • DIVERGENCES.md — new row for the two upstream test files (append-only blocks). packages/codev-gateway and the TUI view test are fork-only, so untracked there.
  • CHANGELOG.md — Unreleased → Provider setup, including the trade-off and the reconfigure note.

🤖 Generated with Claude Code

…dels

A config model entry with no `modalities` resolves image input to false, so
ProviderTransform.message replaced an attached image with an "ERROR: Cannot
read image" note before the request and the model answered that it couldn't
see anything — even when the endpoint supports vision.

Both blocks written by @codev/gateway (netGate configure flow and the custom
provider flow) now declare `attachment: true` and
`modalities: { input: ["text", "image"], output: ["text"] }` per model,
mirroring codev-hub PR #219 so hub-written and self-configured codev.json
stay identical. Every gateway model is now declared vision-capable including
text-only ones: for those the endpoint rejects or ignores the image rather
than the client silently dropping it.

Tests cover the two upstream paths this depends on — config modality →
model capability, and the part filtering in transform — plus the end-to-end
PATCH body from the custom-provider TUI flow. That view test was already
failing on main (it waits on the "Provider id" prompt renamed to "Provider
name" in #137, and predates the provider `name` field); repaired here since
it asserts the payload this changes.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@quickbeard
quickbeard merged commit 8781925 into main Jul 30, 2026
0 of 5 checks passed
@quickbeard
quickbeard deleted the declare-image-modality branch July 30, 2026 10:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant