Repository navigation
[AI Gateway] Document prompt caching on OpenAI-compatible endpoints - #35246
Draft
samscott89 wants to merge 2 commits into
Draft
samscott89 wants to merge 2 commits into
samscott89 wants to merge 2 commits into
Conversation
Add a Prompt caching page. It explains how AI Gateway converts the OpenAI prompt caching fields to Anthropic cache breakpoints, and how to read cache reads and writes in the response usage. Link to the page from the response caching page.
Add a settings table for the OpenAI caching fields and the mode values. State that prompt_cache_breakpoint applies in every mode, and that explicit mode only stops the AI Gateway breakpoint. Shorten sentences and paragraphs, make lists parallel, use the ALL_CAPS placeholder format, and write "1-hour" for the lifetime.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a Prompt caching page for AI Gateway. The existing docs cover response caching only. Users asked how to use Anthropic prompt caching through the OpenAI-compatible endpoints (
/ai/v1/chat/completions,/ai/v1/responses, and/compat/chat/completions).The page covers:
prompt_cache_options.mode: "implicit"prompt_cache_breakpoint, and the Anthropiccache_controlalternativeusageThe caching page now links to the new page.
Do not merge yet. The page says that
mode: "implicit"adds a breakpoint on every Anthropic request. That AI Gateway change is not yet released to production. I will mark this PR ready after the release.Documentation checklist