Prompt Caching is an API-level optimization technique that lets an identical prompt prefix be processed only once across multiple requests, with subsequent calls directly reusing the cached result, thereby substantially reducing cost and latency. Both Anthropic and OpenAI already offer this feature in their APIs.
Grab It in One Sentence First
Prompt Caching turns "re-reading a long document every time" into "read it once, cache it, and then jump straight to the new content."
If your application sends the same system prompt, the same manual, or the same piece of code to the model on every request, then without caching the model processes these tokens from scratch each time, and you pay for it every time. With Prompt Caching, this repeated content is billed only the first time and reused afterward, and the cost can drop by 80% to 90%.
How It Works
When a large language model processes input tokens, it internally generates a series of intermediate computation results (called the KV Cache in the Transformer architecture). The principle of Prompt Caching is: if this request's prefix is completely identical to the previous one, the model provider's infrastructure can skip this repeated computation and continue processing the newly added content directly from the cached intermediate state.
The key condition is that the prefix must be completely identical. Even a single differing character invalidates the cache, and the system reprocesses the entire prefix. This means: fixed content such as the system prompt, document content, and few-shot examples must be placed at the very front of the prompt, with the user's current question at the end.
Anthropic's Prompt Caching requires the cached prefix to be at least 1,024 tokens (2,048 for some models); below this threshold, caching won't be triggered. OpenAI's implementation is slightly different, automatically attempting to cache requests over a certain length without requiring developers to mark it explicitly.
Which Scenarios Are Most Worth It
Applications with a long system prompt: If your system prompt contains thousands of words of role setup, rules, and format requirements, caching can substantially lower the cost of every conversation.
Document Q&A: Placing a long document, product manual, or legal file into the prompt, where the user asks questions based on this document and every request carries the same document. Once the cache hits, the cost of transmitting and processing this document is close to zero.
Fixed context in multi-turn conversations: In a conversation system, the history is usually incremental, but you can mark the fixed part (such as the system prompt and early conversation) as the cached prefix and have the model process only the newly added content.
Code review and codebase Q&A: Placing code files into the prompt and allowing the user to ask questions repeatedly. Caching can make each question cost only the token fee for the new question.
Changes in Cost and Latency
Take Anthropic Claude's pricing as an example: a cache hit (cache read) costs about 10% of a normal input token, meaning each hit can save about 90% of the input cost. The request that first writes to the cache (cache write) has an additional fee, about 25% of a normal input, but you only bear it once.
On latency, a cache-hit response is usually also faster, because a large amount of computation is skipped. For requests containing long documents, the latency improvement is especially noticeable.
Considerations When Using It
The cache has a lifespan. Anthropic's cache lives for 5 minutes by default and is cleared after inactivity, so later requests need to rebuild the cache. If the application has low traffic and long intervals between requests, the hit rate will drop noticeably.
Dynamic content isn't suitable for the cached prefix. If the system prompt contains the current time, a user ID, or variables that change each time, the cache will invalidate frequently. Move the dynamic parts to the end of the prompt and cache only the stable prefix.
Order matters: fixed content must come first, dynamic content last. Reversing them makes the cache completely ineffective.
Not all models support it. When a new model is released, you need to separately confirm whether caching is available, and the pricing and parameters may also differ from older versions.