Skip to content

[AI Gateway] Document prompt caching on OpenAI-compatible endpoints - #35246

Draft
samscott89 wants to merge 2 commits into
cloudflare:productionfrom
samscott89:sscott/ai-gateway-prompt-caching
Draft

samscott89 wants to merge 2 commits into
cloudflare:productionfrom
samscott89:sscott/ai-gateway-prompt-caching

Conversation

@samscott89

Copy link
Copy Markdown
Contributor

Summary

Adds a Prompt caching page for AI Gateway. The existing docs cover response caching only. Users asked how to use Anthropic prompt caching through the OpenAI-compatible endpoints (/ai/v1/chat/completions, /ai/v1/responses, and /compat/chat/completions).

The page covers:

  • The recommended setting for mixed OpenAI and Anthropic traffic: prompt_cache_options.mode: "implicit"
  • How AI Gateway adds Anthropic breakpoints when the request has no caching mode
  • Explicit breakpoints with prompt_cache_breakpoint, and the Anthropic cache_control alternative
  • The cache read and cache write fields in the response usage

The caching page now links to the new page.

Do not merge yet. The page says that mode: "implicit" adds a breakpoint on every Anthropic request. That AI Gateway change is not yet released to production. I will mark this PR ready after the release.

Documentation checklist

Add a Prompt caching page. It explains how AI Gateway converts the
OpenAI prompt caching fields to Anthropic cache breakpoints, and how to
read cache reads and writes in the response usage. Link to the page
from the response caching page.
@github-actions github-actions Bot added the product:ai-gateway AI Gateway: https://developers.cloudflare.com/ai-gateway/ label Nov 15, 2026
Add a settings table for the OpenAI caching fields and the mode
values. State that prompt_cache_breakpoint applies in every mode, and
that explicit mode only stops the AI Gateway breakpoint.

Shorten sentences and paragraphs, make lists parallel, use the
ALL_CAPS placeholder format, and write "1-hour" for the lifetime.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

product:ai-gateway AI Gateway: https://developers.cloudflare.com/ai-gateway/ size/m

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant