What is prompt caching? Cheaper, faster repeated prompts
Prompt caching lets an LLM API reuse work on a prompt prefix it has already processed, cutting input cost and latency. How it works, prices, and a Python demo.

Prompt caching is an LLM API feature that saves the model’s processed state for the start of a prompt (the prefix) and reuses it when a later request begins with exactly the same tokens. Cached input tokens are billed at a fraction of the normal rate, typically a tenth or less, and the response starts faster because the model skips reprocessing them.
It only works on exact prefixes. Change one token near the start of your prompt, such as a timestamp in the system prompt, and everything after it is a cache miss. For agents, which resend their whole history on every turn, that one rule decides most of the bill.
How does prompt caching work?
A transformer processes your prompt into internal key and value tensors for every token, the KV cache, before it writes the first output token. OpenAI’s prompt caching guide describes prompt caching as preserving that state for a reusable prefix: “When a later request has the same prefix and finds a matching cache entry, the model can reuse the saved state instead of processing those tokens again.”
So the provider needs two things to give you a hit: the same model, and a byte-identical prefix. Anthropic’s prompt caching docs spell out the order prefixes are built in: “tools, system, then messages”. Anything that changes in the tool list invalidates the system prompt and messages after it; anything that changes in the system prompt invalidates the messages.
That’s why every caching guide says the same thing in different words: stable content first, variable content last. Tools and instructions at the top, then long documents, then the conversation, and the new user message at the very end.
How much does prompt caching save?
The two big providers price it differently. Prices below are from their docs, checked October 31, 2026.
Anthropic charges for cache writes and discounts reads. A 5-minute cache write costs 1.25x the base input price, a 1-hour write costs 2x, and reads cost 0.1x for most models (0.05x on Claude Opus 5.5 and Sonnet 5.5, 0.025x on Fable 5.1 and Mythos 5.1). For Sonnet 5.5 that’s $2 per million input tokens, $2.50 for a 5-minute write and $0.10 for a read. The 5-minute lifetime refreshes for free each time the entry is used. You can mark up to four breakpoints with cache_control, or set one cache_control at the top level of the request for automatic caching.
OpenAI caches automatically, with no code change. When it launched Prompt Caching in the API, cached tokens on prompts longer than 1,024 tokens got a 50% discount. The current docs differ by model: for GPT-5.6 and later, writes cost 1.25x the input rate for the default 30-minute lifetime (2.5x for 12 hours) and reads cost 0.1x, or 0.05x on GPT-6.1 Sol and Terra; earlier models have no write charge and a model-specific cached-input rate. OpenAI’s own arithmetic: “one write and nine full reads cost 2.15x” the price of one uncached prompt, “compared with 10x without caching.”
Latency drops too. Anthropic’s launch post claimed cost reductions of up to 90% and latency reductions of up to 85% for long prompts, with one example, chatting with a 100,000-token book, going from 11.5 seconds to 2.4 seconds time to first token. I unpack time to first token in LLM latency explained.
What breaks prompt caching?
Here’s a toy prefix cache that hashes each prefix of a block list, run over a 40-turn agent session in three layouts: a stable prefix, a timestamp in the system prompt, and a tool list whose order changes between calls. Prices are Sonnet 5.5’s.
# cache_sim.py: why prompt caching hits or misses, and what it does to the bill.
# A toy prefix cache: hash every prefix of the block list; a request reuses the
# longest prefix written before. Prices: Claude Sonnet 5.5, USD per 1M tokens,
# Anthropic prompt caching docs, checked 2026-10-31 (5-minute TTL).
import hashlib, random
PRICE = {"input": 2.00, "write": 2.50, "read": 0.10}
class PrefixCache:
def __init__(self):
self.seen = set()
def request(self, blocks): # blocks: list of (label, tokens)
h, hit_tokens, total = hashlib.sha256(), 0, 0
for label, n in blocks:
h.update(f"{label}:{n}".encode()); total += n
if h.hexdigest() in self.seen:
hit_tokens = total # longest matching prefix so far
h2 = hashlib.sha256()
for label, n in blocks: # write every prefix of this request (simplification)
h2.update(f"{label}:{n}".encode()); self.seen.add(h2.hexdigest())
miss = total - hit_tokens
usd = (hit_tokens * PRICE["read"] + miss * PRICE["write"]) / 1e6
return hit_tokens, miss, usd
def agent_run(make_blocks, turns=40):
cache, usd, hits, total = PrefixCache(), 0.0, 0, 0
for t in range(turns):
hit, miss, cost = cache.request(make_blocks(t))
usd += cost; hits += hit; total += hit + miss
return usd, hits / total
TOOLS = [("tool:read_file", 300), ("tool:grep", 250), ("tool:edit", 400), ("tool:bash", 350)]
SYSTEM = ("system", 6_000)
history = lambda t: [(f"turn{i}", 1_500) for i in range(t + 1)]
layouts = {
"stable prefix": lambda t: [SYSTEM, *TOOLS, *history(t)],
"timestamp in system prompt": lambda t: [(f"system@{t}", 6_000), *TOOLS, *history(t)],
"tool order shuffles": lambda t: [SYSTEM, *random.Random(t).sample(TOOLS, 4), *history(t)],
}
no_cache = sum(sum(n for _, n in layouts["stable prefix"](t)) for t in range(40)) * PRICE["input"] / 1e6
print(f"{'no caching at all':<28} ${no_cache:6.2f}")
for name, make in layouts.items():
usd, rate = agent_run(make)
print(f"{name:<28} ${usd:6.2f} cache hit rate {rate:5.1%}")
Output (Python 3.13.11):
no caching at all $ 3.04
stable prefix $ 0.31 cache hit rate 95.6%
timestamp in system prompt $ 3.81 cache hit rate 0.0%
tool order shuffles $ 2.19 cache hit rate 44.1%
The session is made up (a 6,000-token system prompt, four tools, 1,500 new tokens per turn) and my cache is simpler than Anthropic’s, which writes entries only at breakpoints and looks back at most 20 blocks for an earlier one. The pattern still holds:
- Stable prefix: 95.6% of input tokens come from cache and the input bill drops by about 90%, from $3.04 to $0.31.
- Timestamp at the top: every request is a new prefix, so nothing ever hits, and you pay the 1.25x write price on everything. Caching made this bill 25% worse than not caching at all.
- Unstable tool order: the system prompt still hits, but everything after the tools misses unless the shuffle happens to repeat an earlier order. Hit rate falls to 44%.
The third one is a real bug. In Unrolling the Codex agent loop (January 23, 2026), OpenAI writes that its initial MCP support “failed to enumerate the tools in a consistent order, causing cache misses.” The same post explains why caching matters so much for agents: the loop is quadratic in the JSON sent over a session, and “when we get cache hits, sampling the model is linear rather than quadratic.”
What other limits should I know?
Minimum length. Short prompts aren’t cached. On Anthropic the minimum is 512 tokens for the newest models (Opus 5.5, Sonnet 5.5 and others), 1,024 for models like Sonnet 4.5 and 4.6, and 4,096 for Opus 4.5, Opus 4.6 and Haiku 4.5. Below the minimum, the request runs uncached “and no error is returned”. OpenAI’s minimum is 1,024 tokens on GPT-5.6 and later.
Lifetime. Anthropic’s 5-minute lifetime is “measured from the start of the request”, so a response that takes 4 minutes to stream leaves about a minute for the follow-up. For agents waiting on humans, the 1-hour option often pays for itself.
Isolation. Caches are scoped to your organization. OpenAI notes that separate prompt_cache_key values also help prevent “cache-hit probing across users”, where someone measures response speed to learn whether a prompt was seen before.
Verify, don’t assume. Anthropic returns cache_creation_input_tokens and cache_read_input_tokens in usage; OpenAI returns cached_tokens. If those are zero in production, your prefix isn’t stable.
What I would do
I’d treat the prefix as a contract. System prompt, tool definitions and long reference documents go first and change only on deploy; anything per-request (dates, user names, retrieved snippets, the question) goes after the last breakpoint. I’d sort tool lists deterministically and freeze them for the session, and keep the date out of the system prompt unless the task needs it, putting it in the user turn instead.
Then I’d log cache read and write tokens per request on a dashboard next to cost, and alert when the hit rate drops after a deploy. A prompt change that breaks caching won’t show up in quality evals, only in the bill. For long agent runs, I’d combine this with resets, as in context inertia, where I priced a 300-turn run with cache reads. And for designing what goes in the prefix in the first place, context engineering is the companion read; the agent harness is where the cache discipline actually lives.
FAQ
Is prompt caching the same as caching responses?
No. A response cache returns a stored answer for an identical request. Prompt caching only reuses the processed input; the model still generates a fresh response every time, so outputs can differ.
Does prompt caching change the model’s output?
It shouldn’t. The cached state is the same computation the model would do on the prefix anyway, and both providers present it as a cost and latency feature, not a quality one.
Do I need to change my code to use prompt caching?
On OpenAI, no: it’s automatic for long enough prompts. On Anthropic, add cache_control at the top level of the request for automatic caching, or on specific content blocks for explicit breakpoints.
How long does a prompt cache last?
On Anthropic, 5 minutes by default, refreshed on each use, or 1 hour at a higher write price. On OpenAI it depends on the model: a 30-minute default (or 12 hours) on GPT-5.6 and later, and roughly 5 to 10 minutes of inactivity for in-memory retention on older models.
Sources
- Anthropic, Prompt caching, docs and pricing table (read October 31, 2026)
- Anthropic, Prompt caching with Claude, launch post (read October 31, 2026)
- OpenAI, Prompt caching guide (read October 31, 2026)
- OpenAI, Prompt Caching in the API, launch post (read October 31, 2026)
- OpenAI, Unrolling the Codex agent loop, January 23, 2026