What to Cache — and What Never to Cache — in an LLM-Backed Feature

Caching feels like a natural fit for any expensive LLM call, but applied carelessly it turns a probabilistic system into a source of silent, stale certainty. Knowing what belongs behind a cache — and what should never be cached — matters more than the caching mechanism itself.

What's safe to cache

The safest candidates are calls whose inputs and outputs are effectively deterministic in the ways that matter to your product: intent classification on a fixed input, embedding generation for a piece of text that won't change, or a lookup-style completion where the answer is stable regardless of exact prompt wording. Cache these aggressively, keyed on a normalized version of the input rather than the raw string — normalize whitespace, casing, and any timestamp or user-ID noise that would otherwise make identical requests miss the cache.

Also cache anything derived purely from static context: a summary of a document that hasn't changed, a generated description of a fixed schema, boilerplate scaffolding the model produces the same way every time. These are effectively pure functions of their input, and treating them as such is correct.

What should never be cached

The failure mode people miss is caching output that depends on information the cache key doesn't capture. A support-triage prompt that references "recent account activity" will change its correct answer as that activity changes, even if the literal prompt text is identical — cache it and you'll serve a stale triage decision with full confidence. Anything involving current state, personalization that updates over time, or a live tool result baked into the prompt is a poor caching candidate unless the cache key includes a fingerprint of that state, not just the prompt text.

Be equally careful with anything safety- or policy-sensitive. A cached refusal or cached approval can outlive the reason it was correct — a moderation call that approved content yesterday under one policy version shouldn't silently keep approving it after the policy changes. Tie these cache entries to a policy version, not just the calendar.

The other trap is caching for cost savings on calls that are cheap to begin with and expensive to get wrong. A misrouted cache hit on a fraction-of-a-cent call that then drives an incorrect user-facing decision isn't a good trade, even though the token savings look attractive on a dashboard.

A useful test before adding a cache layer: write down, in one sentence, what would have to change in the world for this cached answer to become wrong. If you can't name that condition, or can't put it in the cache key, you don't have a caching strategy — you have a bug waiting for the right input.