Prompt caching promises cheaper and faster LLM calls by reusing the processing of a prompt prefix you have already sent. Teams turn it on, check the bill a week later, and see almost no change. Almost always the cause is not the feature but the prompt: it was never structured to be cached.

The concrete details below come from Anthropic's prompt-caching documentation. Other providers differ in thresholds, lifetimes and prices, so check yours, but the principles carry over.

How it works

You mark a point in the prompt as a cache breakpoint with a cache_control field. Everything up to that point is the prefix, and a later request that begins with the identical prefix can read it from cache instead of reprocessing it. The docs describe two ways to enable it: automatic caching, with one top-level cache_control that moves the breakpoint forward as a conversation grows, or explicit breakpoints on individual content blocks, with up to four per request.

The cache is built in a fixed order: tools, then system, then messages. Each level builds on the one before, and a change at one level invalidates that level and everything after it.

The economics

Per the docs, the multipliers on the base input-token price are:

  • Cache write: 1.25× for the default 5-minute lifetime, 2× for the optional 1-hour lifetime.
  • Cache read: 0.1× of the base price for most models, and lower for some of the newest.

So the first request costs a little more than normal, and every request that hits the cache costs a fraction. Caching pays off when you reuse a prefix often enough to recover the write premium. A prefix used once is a small loss.

Why hit rates disappoint

1. The prefix isn't identical. Matching is exact, not semantic. A timestamp in the system prompt, a user ID interpolated into an instruction, or a reordered list of tools changes the tokens, and everything after the change is a miss. The fix is structural: put everything static first (instructions, examples, tool definitions) and everything per-request after the breakpoint.

2. Something upstream invalidates it. The docs list what breaks the cache. Changing tool definitions (names, descriptions, parameters) invalidates the entire cache. Toggling features that modify the system prompt, such as web search or citations, invalidates the system and message caches. Adding or removing images invalidates the message cache. If your framework regenerates a tool schema per request, or serializes tools in a nondeterministic order, you can be missing the cache on every single call without noticing.

3. The prompt is too short. There is a minimum cacheable length, and it varies by model, from 512 tokens on some to 4,096 on others in the current docs. A shorter prompt is not cached even if you mark it, and, importantly, you get no error. It just silently isn't cached.

4. The cache expired. The default lifetime is 5 minutes, refreshed at no extra cost each time the cache is used. A busy feature never notices. A low-traffic tool or a job with long gaps between calls lets the entry expire between uses. The 1-hour option exists for prompts reused less often than every 5 minutes but more than hourly, at the higher write price.

Measure it, don't guess

Every response reports cache behaviour in its usage block:

  • cache_read_input_tokens: tokens served from cache.
  • cache_creation_input_tokens: tokens written to cache.
  • input_tokens: tokens after the last breakpoint, which are not cached.

Total input is the sum of the three. Log all three per request and chart the ratio of reads to total. A sudden drop is the earliest warning that a code change moved something dynamic above the breakpoint. The bill tells you about it weeks later.

A short checklist

  1. Order the prompt: tools, then system, then static context, then the breakpoint, then anything per-request.
  2. Make tool serialization deterministic, and don't edit tool definitions casually. This matters for any setup with many tools, including those built on MCP.
  3. Check your prompt length against the model's minimum.
  4. Choose the lifetime to match your traffic gaps.
  5. Alert on cache-read ratio, not only on cost.

The takeaway

Prompt caching is not a switch you flip. It rewards prompts built as a frozen prefix plus a variable tail, and it punishes casual edits above the breakpoint. Treat the ordering of your prompt as an interface, and measure the hit rate directly.

Sources

Developer ToolsLarge Language ModelsSoftware Engineering