Add vision support to LlamaLanguageModel via mtmd

[james-333i]

Image segments threw unsupportedFeature because the backend had no multimodal path, even though the prebuilt llama.cpp binaries ship the mtmd library. This accepts an mmprojPath at initialization, replaces image segments with the mtmd media marker during prompt formatting, and evaluates text and image chunks through mtmd_helper_eval_chunks before sampling continues. Both respond and streaming support images, and models without a projector keep rejecting image input. Stacks on #195; the squash-merge will sort out the shared commit.

[mattt]

Hi @james-333i. Thanks for these four — I read the whole stack today. #195 is in, so this needs a rebase first (main moved in LlamaLanguageModel.swift with #205). Notes per PR, all small except one:

• Add vision support to LlamaLanguageModel via mtmd #213: would you honor the threads option in mtmdParams.n_threads, and leave a comment that the mtmd_helper_bitmap_init_from_buf signature is pinned to the current llama.swift (master already adds an options argument)?
• Render the Gemma 4 chat format when template detection fails #214: renderGemma4Prompt can't be unit-tested as a private instance method. Could you make it static (or internal) and add a test that renders a short transcript?
• Reuse a per-session context across chat exchanges #215: this is the one I want to talk through. LlamaLanguageModel is @unchecked Sendable with no lock, and acquireSessionContext frees the cached context when a different session comes in. Two sessions sharing one model could free a context the other is still decoding on, where before each generation owned its own. Either a lock around the cache or a documented one-generation-at-a-time rule would do; which would you prefer? Also worth a line in the clearCachedContext() docs that the context (and its KV) now lives as long as the model.
• Add tool calling to LlamaLanguageModel, including streaming #216: this bundles @noorbhatia's Support tool calling in MLXLanguageModel.streamResponse() #181, so it waits on the Make LanguageModelSession transcript observable while streaming responses #103 decision. Two things on your side meanwhile: the Linux legs fail on CFGetTypeID/CFBooleanGetTypeID in LlamaToolCallFormat.swift (not in swift-corelibs-foundation), and the last streamed snapshot can shrink (the .stop path yields emittedBase + roundVisible, then the post-loop yield goes back to emittedBase). Stopping at the first </tool_call> also means only one call per round; if that's intentional, say so in the docs.

The LlamaToolCallFormat tests are great — thank you for making that part pure.

[james-333i]

@mattt Thanks for the thorough review. All four are rebased onto main and updated.

#213: n_threads now follows the model's threads setting, and the bitmap helper call carries a note that it is pinned to the current llama.swift signature.

#214: renderGemma4Prompt is static and internal now, with a small suite rendering a short transcript and the assistant prefill.

#215: I went with the lock, but with checkout semantics rather than a bare mutex, since a lock around the accessor alone would not protect a context for the length of a generation. A generation checks the cached context out for its whole run. A concurrent generation for another session gets a transient context that is never cached, so nothing can free a context that is still decoding. clearCachedContext() during a run marks the context to be freed on release. The docs now say the context and its KV live as long as the model.

#216: the boolean check is Darwin-conditional with an objCType fallback for corelibs-foundation, and the last snapshot no longer shrinks: the round's visible text is folded into the base before the .stop yield. One call per round is intentional and now documented on callTerminator. Models that want several calls issue them across consecutive rounds. Happy to widen it to multiple calls per round as a follow-up if you would rather have that.

[Copilot]

🟡 Changes recommended

Structured image generation and concurrent mtmd evaluation remain unsafe.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Adds mtmd-based image support to the local Llama backend.

Changes:

• Adds optional multimodal projector loading and image prompt formatting.
• Supports image generation in synchronous and streaming responses.
• Adds vision integration tests.

File summaries

File	Description
LlamaLanguageModel.swift	Loads mtmd projectors and evaluates image prompts.
LlamaLanguageModelTests.swift	Tests image responses, streaming, and projector rejection.

Review details

• Files reviewed: 2/2 changed files
• Comments generated: 3
• Review effort level: Balanced

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[mattt]

Hi @james-333i. Thanks for the rebase!

Two things on this one before I merge it, and one nit:

1. Structured output with images (session.respond(to:images:generating:)) doesn't go through the projector. With includeSchemaInPrompt on, the schema branch uses the text-only formatter and throws unsupportedFeature even when a projector is loaded; with it off, the marker gets tokenized as plain text and the model answers about an image it never saw. I'd be happy with an explicit unsupportedFeature in both configurations for now (image-aware formatter in the schema branch, then guard on promptImages.isEmpty before the JSON generator), with the full mtmd prefill + constrained sampling as a follow-up.
2. mtmd_helper_eval_chunks is documented as not thread-safe, and it runs on the one shared mtmdContext. The text path shares only the immutable model and makes a context per call, so this is the first piece of mutable native state that's touched during generation. Could you hold an NSLock from bitmap init through eval_chunks, released before the sampling loop?
3. Nit: rejectsImagesWithoutProjector can assert LlamaLanguageModelError.unsupportedFeature directly; the enum is Equatable for free.

While you're in there, two small ones on #215: the catch in generateChatText discards the cached context even when the failing generation ran on a transient (guard it on cachedSessionContext?.context == context), and lastReusedTokenCount/lastPrefillTokenCount are written outside the lock. Then I'll merge #213, #214, and #215 in order.

[james-333i]

@mattt Thank you. All five are in and the stack is re-pushed.

#213. The schema branch now uses the image-aware formatter. The structured path throws unsupportedFeature whenever images are present with the schema in the prompt or not. The mtmd prefill with constrained sampling can follow as a separate PR. Bitmap init through mtmd_helper_eval_chunks now runs under a lock. The lock is released before the sampling loop. The test asserts unsupportedFeature directly.

#215. The catch only discards the cache when the failing generation ran on the cached context. The two diagnostic counters are written under the lock.