Most LLM providers offer some form of prompt caching: if the beginning of a request matches a recently processed prefix exactly, the provider can reuse earlier work, which usually means lower cost and lower latency on the cached portion. The details differ by provider (minimum prefix lengths, expiry windows, whether you mark cache breakpoints explicitly), so check your provider's documentation. But the underlying principle is the same everywhere, and it is a prompt-layout problem you control.

The rule: cache hits are prefix matches

A cache can only reuse a prefix. The moment one token differs, everything after that point is processed fresh. That makes the *order* of your prompt a performance decision:

  1. System instructions and policies — change rarely, so they go first.
  2. Tool definitions and output schemas — also stable between deploys.
  3. Few-shot examples and reference documents — stable per task or per tenant.
  4. Conversation history — grows by appending, so earlier turns stay a valid prefix.
  5. The current user input and any per-request data — different every time, so it goes last.

The common mistake is putting something volatile near the top. A timestamp in the first line ("Current time: 14:32:07"), a request ID, a user's name interpolated into the opening sentence, or a randomly shuffled list of examples will each invalidate the cache for everything beneath them. If the model needs the current date, put it in the final user-turn section, not the system prompt.

Other ways prefixes quietly stop matching

  • Non-deterministic serialization. If you render tool schemas or JSON context from a dictionary and key order varies between processes or versions, the text differs even though the content doesn't. Sort keys, or render from a fixed template.
  • Editing history. Trimming or summarizing old turns on every request changes the prefix each time. If you must compact history, do it in occasional batches (for example, when you cross a size threshold) rather than sliding the window one message at a time.
  • Per-user personalization in the system prompt. Moving user-specific facts out of the shared prefix and into a later section lets all users share the same cached head.
  • Whitespace and template drift. A trailing newline added in a refactor is a cache miss on every request until entries expire.

How to know it's working

Do not assume. Most APIs report how many input tokens were served from cache in the response usage metadata. Log that figure per request, compute a hit rate per prompt template, and alert when it drops after a deploy. A sudden drop is usually a volatile value that slipped into the prefix.

Pair this with the golden-file habit: snapshot the rendered prompt for two different sample requests and diff them. The first position where they diverge is your effective cacheable prefix. If that position is in the first few lines, you have found your problem.

The takeaway: treat the top of your prompt as a shared, versioned artifact and the bottom as the only place per-request data is allowed to live. It costs nothing to arrange this way up front, and it is tedious to untangle after a prompt has accumulated months of edits.

Large Language ModelsSoftware Engineering