Skip to content

Add a Core AI backend and a Private Cloud Compute model - #218

Draft
james-333i wants to merge 2 commits into
huggingface:mainfrom
james-333i:feat/coreai-backend
Draft

Add a Core AI backend and a Private Cloud Compute model#218
james-333i wants to merge 2 commits into
huggingface:mainfrom
james-333i:feat/coreai-backend

Conversation

@james-333i

Copy link
Copy Markdown
Contributor

Follow-up to the discussion in #210. This adds two Foundation Models protocol backends behind the OS 27 LanguageModelSession(model:) construction.

  • CoreAILanguageModel (new CoreAI trait, wired like the MLX and Llama traits) runs Core AI .aimodel bundles exported with apple/coreai-models recipes. Verified on device (iPhone 17 Pro, iOS 27 beta) with Qwen3 and SmolLM2 exports: streaming, multi-turn transcripts, and the executor's tool-calling path.
  • PrivateCloudComputeLanguageModel wraps the OS 27 FoundationModels.PrivateCloudComputeLanguageModel, exposing availability, quota, and vision and tool capabilities. It requires the com.apple.developer.private-cloud-compute entitlement, which Apple grants on request for apps under two million users. Verified on device, including tool calling.

To share the session plumbing, the FM adapter's respond and streamResponse bodies move verbatim into file-scope fmRespond and fmStreamResponse helpers parameterized by an async session factory. All three backends delegate to them. Streamed and returned text also drops leading newlines, which OS 27 executors emit when they strip an empty reasoning block from the output.

Draft until the dependency is upstream. The CoreAI trait currently depends on a fork of apple/coreai-models that carries two fixes offered upstream. Links are tracked in #210.

Assuming those get approved (or similar fixes are issued by Apple), the dependency flips to apple/coreai-models and this becomes mergeable. This branch includes the fix from the standalone duplicated-trailing-prompt PR and will be rebased once that merges.

james-333i and others added 2 commits September 3, 2026 20:09
LanguageModelSession appends the prompt entry to its transcript before
invoking the model. The Foundation Models adapter then passes that
transcript to FoundationModels.LanguageModelSession and also sends the
prompt through respond/streamResponse, which appends it again, so every
FM-backed turn reaches the model with the final user message duplicated.

The MLX adapter shows the intended contract: it renders the chat from
session.transcript alone and treats the prompt parameter as a fallback.
FM's API requires passing the prompt separately, so the fix on this
adapter is to drop a trailing transcript prompt that matches the
outgoing prompt before converting.

Verified by decoding the rendered prompt of a custom LanguageModel
backend on OS 27, where the doubled turn is directly observable; the
same transcript construction applies to SystemLanguageModel on OS 26.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Adds two Foundation Models protocol backends behind the OS 27 session
machinery:

- CoreAILanguageModel (new CoreAI trait): runs Core AI .aimodel bundles
  exported with apple/coreai-models recipes through the new
  LanguageModelSession(model:) construction, mirroring the MLX and Llama
  trait wiring. Verified on device (iPhone 17 Pro, iOS 27 beta) with
  Qwen3 and SmolLM2 exports: streaming, multi-turn transcripts, and the
  executor's tool-calling path.
- PrivateCloudComputeLanguageModel: wraps the OS 27
  FoundationModels.PrivateCloudComputeLanguageModel (availability,
  quota, vision/tool capabilities). Requires the
  com.apple.developer.private-cloud-compute entitlement, granted on
  request for apps under two million users. Verified on device including
  tool calling.

To share the session plumbing, the Foundation Models adapter's
respond/streamResponse bodies move verbatim into file-scope
fmRespond/fmStreamResponse helpers parameterized by an async session
factory; SystemLanguageModel, the Core AI backend, and the PCC backend
all delegate to them. Streamed and returned text also drops leading
newlines, which OS 27 executors emit when they strip an empty reasoning
block from the output.

The CoreAI trait depends on a fork of apple/coreai-models because the
upstream package cannot be consumed yet by lower-deployment-target apps
or built for the simulator, and its marker detection breaks unk-token
tokenizers. All three fixes are offered upstream; this PR stays a draft
until they land and the dependency can point at apple/coreai-models.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mattt
mattt marked this pull request as draft September 4, 2026 12:46
@mattt

mattt commented Sep 4, 2026

Copy link
Copy Markdown
Collaborator

Thanks, @james-333i. I've marked this as a draft until the two apple/coreai-models fixes land, as your description says. I'll read through the shape in the meantime.

@james-333i

Copy link
Copy Markdown
Contributor Author

@mattt Since filing this I have found a method that removes the dependency on apple/coreai-models entirely, so I would like to restructure the PR before you spend time on the current one.

iOS 27's FoundationModels defines a public LanguageModel protocol, and LanguageModelSession.init(model:) accepts any conformer. The Core AI models in apple/coreai-models already conform to it and so does Private Cloud Compute. Reading this PR's adapter with that in mind, the only Core AI specific lines are the construction of the model. Everything else hands it to FoundationModels.LanguageModelSession(model:) through the shared bridge.

Proposed integration: a generic wrapper for any FoundationModels.LanguageModel conformer, taking an async factory so heavy loading stays lazy and the consumer owns load and unload. It depends only on the FoundationModels SDK framework behind @available(iOS 27, macOS 27, *). No CoreAI trait, no coreai-models dependency, no fork branch, nothing hosted. Core AI becomes a documented recipe: add apple/coreai-models to your app (the source package if your deployment floor is 27, or a self-built xcframework if you support earlier OS versions, which I have verified works with library evolution and CoreAI weak-linked), construct the model, and hand it to the wrapper. The PCC adapter and the shared bridge refactor stay as they are.

That makes this mergeable without waiting on anything from Apple, and it covers any future model that adopts Apple's protocol. If that works for you I will push the restructured branch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants