From 81ffdee3ef565da5f337a5656d370b4ca63912ef Mon Sep 17 00:00:00 2001 From: Sam Scott Date: Sun, 15 Nov 2026 10:24:17 -0600 Subject: [PATCH 1/2] [AI Gateway] Document prompt caching on OpenAI-compatible endpoints Add a Prompt caching page. It explains how AI Gateway converts the OpenAI prompt caching fields to Anthropic cache breakpoints, and how to read cache reads and writes in the response usage. Link to the page from the response caching page. --- .../docs/ai-gateway/features/caching.mdx | 2 + .../ai-gateway/features/prompt-caching.mdx | 174 ++++++++++++++++++ 2 files changed, 176 insertions(+) create mode 100644 src/content/docs/ai-gateway/features/prompt-caching.mdx diff --git a/src/content/docs/ai-gateway/features/caching.mdx b/src/content/docs/ai-gateway/features/caching.mdx index 28ce6782a16..52794518ce9 100644 --- a/src/content/docs/ai-gateway/features/caching.mdx +++ b/src/content/docs/ai-gateway/features/caching.mdx @@ -13,6 +13,8 @@ import { Render, TabItem, Tabs } from "~/components"; When caching is enabled, AI Gateway can cache responses from your AI model providers, serving them directly from Cloudflare's cache for identical requests. +To reuse prompt prefixes at the provider instead, refer to [Prompt caching](/ai-gateway/features/prompt-caching/). + ## Benefits of Using Caching - **Reduced Latency:** Serve responses faster to your users by avoiding a round trip to the origin AI provider for repeated requests. diff --git a/src/content/docs/ai-gateway/features/prompt-caching.mdx b/src/content/docs/ai-gateway/features/prompt-caching.mdx new file mode 100644 index 00000000000..551dbf10bf1 --- /dev/null +++ b/src/content/docs/ai-gateway/features/prompt-caching.mdx @@ -0,0 +1,174 @@ +--- +pcx_content_type: configuration +title: Prompt caching +sidebar: + order: 3 +description: Reuse prompt prefixes on OpenAI and Anthropic models through the OpenAI-compatible endpoints of AI Gateway. +products: + - ai-gateway +--- + +Prompt caching lets a model provider reuse the processed start of a prompt. Requests that share a long, stable prefix cost less and return faster. + +Prompt caching is different from AI Gateway [response caching](/ai-gateway/features/caching/). Response caching stores a full response and returns it for an identical request. Prompt caching happens at the provider, and the model still generates a new response. + +## How providers cache prompts + +OpenAI caches prompt prefixes automatically. Anthropic caches a prefix only when the request marks where the prefix ends. Anthropic calls this mark a cache breakpoint and sets it with `cache_control`. + +When you send an OpenAI-format request to an Anthropic model, AI Gateway converts the request to the Anthropic Messages API. During the conversion, AI Gateway reads the OpenAI prompt caching fields and adds the Anthropic breakpoints for you. + +This applies to these endpoints: + +- `/ai/v1/chat/completions` and `/ai/v1/responses` on the [REST API](/ai-gateway/usage/rest-api/) +- `/compat/chat/completions` on the [Unified API](/ai-gateway/usage/chat-completion/) + +## Cache prompts on OpenAI and Anthropic models + +To cache prompts with one request body for both providers, set `prompt_cache_options.mode` to `implicit`: + +```bash title="Request with implicit prompt caching" +# Run `wrangler whoami` to get your account ID to replace $CLOUDFLARE_ACCOUNT_ID, +# and `wrangler auth token` to get an auth token to replace $CLOUDFLARE_API_TOKEN. +curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ID/ai/v1/chat/completions" \ + --header "Authorization: Bearer $CLOUDFLARE_API_TOKEN" \ + --header "Content-Type: application/json" \ + --data '{ + "model": "anthropic/claude-sonnet-5", + "messages": [ + { + "role": "system", + "content": "" + }, + { + "role": "user", + "content": "What is Cloudflare?" + } + ], + "prompt_cache_options": { + "mode": "implicit" + } + }' +``` + +To send the same request to OpenAI, change `model` to an OpenAI model, such as `openai/gpt-4.1`. The rest of the body does not change. + +- **OpenAI models** receive `prompt_cache_options` unchanged. +- **Anthropic models** receive a top-level `cache_control` breakpoint on every request. Anthropic uses this breakpoint to cache the prompt prefix automatically. + +Put content that does not change, such as instructions and tool definitions, at the start of the request. Put content that changes at the end. + +:::note +Anthropic does not cache a prompt that is shorter than the minimum length for the model. The request still succeeds. For the minimum lengths, refer to [Prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) in the Anthropic documentation. +::: + +### Set the cache lifetime + +To change how long Anthropic keeps a cache entry, set `prompt_cache_options.ttl`. AI Gateway maps the OpenAI lifetime to the nearest Anthropic lifetime: + +| OpenAI `ttl` | Anthropic lifetime | +| ------------ | ------------------ | +| `30m` | 5 minutes | +| `12h` | 1 hour | + +A longer lifetime costs more to write. Use `12h` only when you reuse a prefix after more than five minutes. OpenAI accepts different `ttl` values on Chat Completions and on Responses. For the values each endpoint accepts, refer to [Prompt caching](https://platform.openai.com/docs/guides/prompt-caching) in the OpenAI documentation. + +## Requests without a caching mode + +If a request does not set `prompt_cache_options.mode`, AI Gateway adds an Anthropic breakpoint only when the prefix is likely to be read again. This is true when the request has one of these signals: + +- A `prompt_cache_key` field +- A `cf-aig-session-id` header. For more information, refer to [Session affinity](/ai-gateway/features/routing/auto-router/#session-affinity). +- An earlier assistant message or tool result, which shows that the request continues a conversation + +A first request with no signal gets no breakpoint. Anthropic charges extra to write a cache entry, so a one-time request would pay for an entry that no request reads. + +## Place breakpoints yourself + +To control where Anthropic breakpoints go, set `prompt_cache_options.mode` to `explicit`. Then add `prompt_cache_breakpoint` to each text part or tool that ends a prefix to cache: + +```json title="Request with an explicit breakpoint" +{ + "model": "anthropic/claude-sonnet-5", + "messages": [ + { + "role": "system", + "content": [ + { + "type": "text", + "text": "", + "prompt_cache_breakpoint": {} + } + ] + }, + { + "role": "user", + "content": "What is Cloudflare?" + } + ], + "prompt_cache_options": { + "mode": "explicit" + } +} +``` + +AI Gateway converts each `prompt_cache_breakpoint` to a `cache_control` breakpoint on the same part. In `explicit` mode, AI Gateway adds no other breakpoints. + +These limits apply: + +- Anthropic accepts up to four breakpoints in a request. AI Gateway removes the extra breakpoints, starting with the oldest message. +- Images, files, and audio cannot carry a breakpoint. AI Gateway ignores a breakpoint on these parts. +- Anthropic requires 1-hour breakpoints to come before 5-minute breakpoints. AI Gateway shortens a 1-hour breakpoint that comes after a 5-minute breakpoint. + +### Use Anthropic cache_control + +If you send a request only to Anthropic models, you can add Anthropic `cache_control` markers to content parts and tools. AI Gateway sends these markers as written. + +When any part of a request has `cache_control`, AI Gateway ignores `prompt_cache_options` and `prompt_cache_breakpoint` in that request. Do not mix the two styles in one request. + +To use Anthropic prompt caching with no conversion, send requests to the `/ai/v1/messages` endpoint. For more information, refer to [REST API](/ai-gateway/usage/rest-api/). + +## Read cache usage + +The `usage` object in the response reports cache reads and cache writes. The input token count includes the cached tokens and the cache write tokens. + +The fields depend on the endpoint: + +| Value | Chat Completions field | Responses field | +| --------------------------------- | --------------------------------------------------- | ----------------------------------------------- | +| Tokens read from the cache | `usage.prompt_tokens_details.cached_tokens` | `usage.input_tokens_details.cached_tokens` | +| Tokens written to the cache | `usage.prompt_tokens_details.cache_write_tokens` | `usage.input_tokens_details.cache_write_tokens` | +| Tokens written with a 1h lifetime | `usage.prompt_tokens_details.cache_write_1h_tokens` | — | + +When AI Gateway converts a response from an Anthropic model, it adds the cache write fields. A cache write field appears only when its value is greater than zero. For the fields that OpenAI returns, refer to the OpenAI documentation. + +The first request writes the prefix to the cache: + +```json title="Usage on the first request" +{ + "usage": { + "prompt_tokens": 2100, + "completion_tokens": 120, + "total_tokens": 2220, + "prompt_tokens_details": { + "cached_tokens": 0, + "cache_write_tokens": 2048 + } + } +} +``` + +A later request with the same prefix reads it from the cache: + +```json title="Usage on a later request" +{ + "usage": { + "prompt_tokens": 2100, + "completion_tokens": 95, + "total_tokens": 2195, + "prompt_tokens_details": { + "cached_tokens": 2048 + } + } +} +``` From 03b31491b72a5c294c1bd211e3761de406dae9f4 Mon Sep 17 00:00:00 2001 From: Sam Scott Date: Sun, 15 Nov 2026 15:23:09 -0600 Subject: [PATCH 2/2] [AI Gateway] Apply style guide to the prompt caching page Add a settings table for the OpenAI caching fields and the mode values. State that prompt_cache_breakpoint applies in every mode, and that explicit mode only stops the AI Gateway breakpoint. Shorten sentences and paragraphs, make lists parallel, use the ALL_CAPS placeholder format, and write "1-hour" for the lifetime. --- .../ai-gateway/features/prompt-caching.mdx | 99 +++++++++++-------- 1 file changed, 59 insertions(+), 40 deletions(-) diff --git a/src/content/docs/ai-gateway/features/prompt-caching.mdx b/src/content/docs/ai-gateway/features/prompt-caching.mdx index 551dbf10bf1..98b23116a7c 100644 --- a/src/content/docs/ai-gateway/features/prompt-caching.mdx +++ b/src/content/docs/ai-gateway/features/prompt-caching.mdx @@ -8,24 +8,42 @@ products: - ai-gateway --- -Prompt caching lets a model provider reuse the processed start of a prompt. Requests that share a long, stable prefix cost less and return faster. +Prompt caching lets a model provider reuse the processed start of a prompt. Requests that share a long prefix cost less and return faster. -Prompt caching is different from AI Gateway [response caching](/ai-gateway/features/caching/). Response caching stores a full response and returns it for an identical request. Prompt caching happens at the provider, and the model still generates a new response. +Prompt caching is different from AI Gateway [response caching](/ai-gateway/features/caching/). Response caching returns a stored response for an identical request. With prompt caching, the model still generates a new response. ## How providers cache prompts -OpenAI caches prompt prefixes automatically. Anthropic caches a prefix only when the request marks where the prefix ends. Anthropic calls this mark a cache breakpoint and sets it with `cache_control`. +OpenAI caches prompt prefixes automatically. Anthropic caches a prefix only when the request marks where the prefix ends. This mark is a cache breakpoint, set with the `cache_control` field. -When you send an OpenAI-format request to an Anthropic model, AI Gateway converts the request to the Anthropic Messages API. During the conversion, AI Gateway reads the OpenAI prompt caching fields and adds the Anthropic breakpoints for you. +AI Gateway converts OpenAI-format requests to the Anthropic Messages API for Anthropic models. During the conversion, AI Gateway reads the OpenAI prompt caching fields. It then adds the Anthropic breakpoints for you. -This applies to these endpoints: +This conversion applies to the `/ai/v1/chat/completions` and `/ai/v1/responses` endpoints of the [REST API](/ai-gateway/usage/rest-api/). It also applies to the `/compat/chat/completions` endpoint of the [Unified API](/ai-gateway/usage/chat-completion/). -- `/ai/v1/chat/completions` and `/ai/v1/responses` on the [REST API](/ai-gateway/usage/rest-api/) -- `/compat/chat/completions` on the [Unified API](/ai-gateway/usage/chat-completion/) +## Settings + +AI Gateway reads these OpenAI fields when it converts a request for an Anthropic model: + +| Field | Valid values | Effect on Anthropic models | +| --------------------------- | -------------------------------- | --------------------------------------------------------------------------------- | +| `prompt_cache_options.mode` | `implicit`, `explicit` | Controls whether AI Gateway adds its own breakpoint. Refer to the next table. | +| `prompt_cache_options.ttl` | `30m`, `12h` | Sets the lifetime of every breakpoint. Without a value, Anthropic uses 5 minutes. | +| `prompt_cache_breakpoint` | An object with an optional `ttl` | Adds a breakpoint on the text part or tool that carries it. | +| `prompt_cache_key` | A string | Shows that later requests reuse the prefix. | + +The `mode` value controls the breakpoint that AI Gateway adds at the end of the prompt: + +| `mode` | Breakpoint from AI Gateway | +| ---------- | ------------------------------------------------------------------------------------------------------ | +| `implicit` | Added on every request. | +| `explicit` | Never added. Only the parts with `prompt_cache_breakpoint` get a breakpoint. | +| Not set | Added only when the request shows reuse. Refer to [Requests without a mode](#requests-without-a-mode). | + +OpenAI models receive these fields unchanged. ## Cache prompts on OpenAI and Anthropic models -To cache prompts with one request body for both providers, set `prompt_cache_options.mode` to `implicit`: +To cache prompts with one request body for both providers, set `prompt_cache_options.mode` to `implicit`. Replace `` with long instructions that are the same on every request: ```bash title="Request with implicit prompt caching" # Run `wrangler whoami` to get your account ID to replace $CLOUDFLARE_ACCOUNT_ID, @@ -38,7 +56,7 @@ curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ "messages": [ { "role": "system", - "content": "" + "content": "" }, { "role": "user", @@ -51,12 +69,9 @@ curl -X POST "https://api.cloudflare.com/client/v4/accounts/$CLOUDFLARE_ACCOUNT_ }' ``` -To send the same request to OpenAI, change `model` to an OpenAI model, such as `openai/gpt-4.1`. The rest of the body does not change. +To send the request to OpenAI, change `model` to an OpenAI model, such as `openai/gpt-4.1`. Do not change the rest of the body. -- **OpenAI models** receive `prompt_cache_options` unchanged. -- **Anthropic models** receive a top-level `cache_control` breakpoint on every request. Anthropic uses this breakpoint to cache the prompt prefix automatically. - -Put content that does not change, such as instructions and tool definitions, at the start of the request. Put content that changes at the end. +Put content that does not change at the start of the request. This includes instructions and tool definitions. Put content that changes at the end. :::note Anthropic does not cache a prompt that is shorter than the minimum length for the model. The request still succeeds. For the minimum lengths, refer to [Prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) in the Anthropic documentation. @@ -64,28 +79,34 @@ Anthropic does not cache a prompt that is shorter than the minimum length for th ### Set the cache lifetime -To change how long Anthropic keeps a cache entry, set `prompt_cache_options.ttl`. AI Gateway maps the OpenAI lifetime to the nearest Anthropic lifetime: +To change how long Anthropic keeps a cache entry, set `prompt_cache_options.ttl`. AI Gateway maps each OpenAI lifetime to the nearest Anthropic lifetime: | OpenAI `ttl` | Anthropic lifetime | | ------------ | ------------------ | | `30m` | 5 minutes | | `12h` | 1 hour | -A longer lifetime costs more to write. Use `12h` only when you reuse a prefix after more than five minutes. OpenAI accepts different `ttl` values on Chat Completions and on Responses. For the values each endpoint accepts, refer to [Prompt caching](https://platform.openai.com/docs/guides/prompt-caching) in the OpenAI documentation. +A longer lifetime costs more to write. Use `12h` only when you reuse a prefix after more than five minutes. + +OpenAI accepts different `ttl` values on Chat Completions and on Responses. For the values each endpoint accepts, refer to [Prompt caching](https://platform.openai.com/docs/guides/prompt-caching) in the OpenAI documentation. -## Requests without a caching mode +## Requests without a mode -If a request does not set `prompt_cache_options.mode`, AI Gateway adds an Anthropic breakpoint only when the prefix is likely to be read again. This is true when the request has one of these signals: +When a request does not set `prompt_cache_options.mode`, AI Gateway adds a breakpoint only if the request shows reuse. A request shows reuse when it has one of these signals: - A `prompt_cache_key` field -- A `cf-aig-session-id` header. For more information, refer to [Session affinity](/ai-gateway/features/routing/auto-router/#session-affinity). -- An earlier assistant message or tool result, which shows that the request continues a conversation +- A `cf-aig-session-id` header +- An earlier assistant message or tool result + +For more information about the session header, refer to [Session affinity](/ai-gateway/features/routing/auto-router/#session-affinity). -A first request with no signal gets no breakpoint. Anthropic charges extra to write a cache entry, so a one-time request would pay for an entry that no request reads. +A first request with no signal gets no breakpoint. Anthropic charges extra to write a cache entry. Without a signal, no later request is likely to read the entry. ## Place breakpoints yourself -To control where Anthropic breakpoints go, set `prompt_cache_options.mode` to `explicit`. Then add `prompt_cache_breakpoint` to each text part or tool that ends a prefix to cache: +To mark the end of a prefix, add `prompt_cache_breakpoint` to a text part or a tool. AI Gateway converts each one to a `cache_control` breakpoint on the same part. + +To use only your own breakpoints, also set `prompt_cache_options.mode` to `explicit`: ```json title="Request with an explicit breakpoint" { @@ -96,7 +117,7 @@ To control where Anthropic breakpoints go, set `prompt_cache_options.mode` to `e "content": [ { "type": "text", - "text": "", + "text": "", "prompt_cache_breakpoint": {} } ] @@ -112,35 +133,33 @@ To control where Anthropic breakpoints go, set `prompt_cache_options.mode` to `e } ``` -AI Gateway converts each `prompt_cache_breakpoint` to a `cache_control` breakpoint on the same part. In `explicit` mode, AI Gateway adds no other breakpoints. - -These limits apply: +These limits apply to breakpoints: -- Anthropic accepts up to four breakpoints in a request. AI Gateway removes the extra breakpoints, starting with the oldest message. +- Anthropic accepts up to four breakpoints in a request. AI Gateway removes extra breakpoints, starting with the oldest message. - Images, files, and audio cannot carry a breakpoint. AI Gateway ignores a breakpoint on these parts. -- Anthropic requires 1-hour breakpoints to come before 5-minute breakpoints. AI Gateway shortens a 1-hour breakpoint that comes after a 5-minute breakpoint. +- Anthropic requires 1-hour breakpoints before 5-minute breakpoints. AI Gateway shortens a 1-hour breakpoint that comes later. -### Use Anthropic cache_control +### Use Anthropic cache_control markers -If you send a request only to Anthropic models, you can add Anthropic `cache_control` markers to content parts and tools. AI Gateway sends these markers as written. +For requests to Anthropic models only, you can add `cache_control` markers to content parts and tools. AI Gateway sends these markers as written. -When any part of a request has `cache_control`, AI Gateway ignores `prompt_cache_options` and `prompt_cache_breakpoint` in that request. Do not mix the two styles in one request. +If any part has `cache_control`, AI Gateway ignores the OpenAI caching fields in that request. Do not mix the two styles in one request. -To use Anthropic prompt caching with no conversion, send requests to the `/ai/v1/messages` endpoint. For more information, refer to [REST API](/ai-gateway/usage/rest-api/). +To use Anthropic prompt caching with no conversion, use the `/ai/v1/messages` endpoint. For more information, refer to [REST API](/ai-gateway/usage/rest-api/). ## Read cache usage -The `usage` object in the response reports cache reads and cache writes. The input token count includes the cached tokens and the cache write tokens. +The `usage` object in the response reports cache reads and cache writes. The input token count includes the cache read and cache write tokens. -The fields depend on the endpoint: +The field names depend on the endpoint: -| Value | Chat Completions field | Responses field | -| --------------------------------- | --------------------------------------------------- | ----------------------------------------------- | -| Tokens read from the cache | `usage.prompt_tokens_details.cached_tokens` | `usage.input_tokens_details.cached_tokens` | -| Tokens written to the cache | `usage.prompt_tokens_details.cache_write_tokens` | `usage.input_tokens_details.cache_write_tokens` | -| Tokens written with a 1h lifetime | `usage.prompt_tokens_details.cache_write_1h_tokens` | — | +| Value | Chat Completions field | Responses field | +| ----------------------------------- | --------------------------------------------------- | ----------------------------------------------- | +| Tokens read from the cache | `usage.prompt_tokens_details.cached_tokens` | `usage.input_tokens_details.cached_tokens` | +| Tokens written to the cache | `usage.prompt_tokens_details.cache_write_tokens` | `usage.input_tokens_details.cache_write_tokens` | +| Tokens written with 1-hour lifetime | `usage.prompt_tokens_details.cache_write_1h_tokens` | — | -When AI Gateway converts a response from an Anthropic model, it adds the cache write fields. A cache write field appears only when its value is greater than zero. For the fields that OpenAI returns, refer to the OpenAI documentation. +AI Gateway adds the cache write fields to converted responses from Anthropic models. A cache write field appears only when its value is more than zero. For the fields that OpenAI returns, refer to the OpenAI documentation. The first request writes the prefix to the cache: