A Practical Walkthrough: Adding Semantic Caching to an LLM Endpoint

Exact-match caching barely helps an LLM-backed endpoint. Real users rarely send the same string twice; they rephrase, add a word, fix a typo, or ask the same underlying question in a different order. If your cache key is a hash of the raw input, you'll watch your hit rate sit near zero while paying full price for requests that are, semantically, repeats. Semantic caching fixes this — but it trades a simple bug (never hitting) for a more dangerous one (hitting when you shouldn't), so it's worth building deliberately.

The mechanics are straightforward. When a request comes in, embed the query and search a vector store for the nearest previously-served queries above some similarity threshold. If you find a close enough match, and its context is still valid, return the cached response instead of calling the model. If not, call the model as usual and store the new query, its embedding, and the response for future lookups.

Picking a similarity threshold is a product decision, not a tuning knob

The threshold is the whole ballgame, and it isn't something you set once from intuition. Too loose, and "what's your refund policy for annual plans" starts serving the cached answer for "what's your refund policy for monthly plans" — a wrong answer delivered with full confidence, which is worse than the latency of a fresh call. Too tight, and you're back to near-zero hit rate with extra infrastructure to show for it.

Build a small labeled set of query pairs that should and shouldn't reuse a cached response, pulled from real near-duplicate traffic if you have it. Sweep the threshold against that set and pick the point that keeps false reuse rate near zero, even if it costs you some hit rate. Recheck this periodically — query patterns drift as the product changes, and a threshold tuned for last quarter's traffic can quietly start admitting bad matches as usage shifts.

Cache invalidation still matters more than people expect

Semantic caching inherits every classic caching problem, plus one specific to LLM responses: a query can be semantically stable while its correct answer isn't. "What's the current usage limit on the free tier" is the same question today and next month, but the right answer can change. Give time-sensitive queries a short TTL rather than treating every cache entry as equally durable, and be explicit about which query types are eligible for caching at all — anything referencing "today," "now," or account-specific state usually shouldn't be.

Personalized or user-scoped queries need a namespaced cache key, not a shared pool. Two users asking "what's my current plan" are not asking the same question even though the text is identical, and a cache that doesn't account for that will happily leak one user's answer to another.

Monitor hit quality, not just hit rate

A cache hit rate dashboard tells you how often you saved a call; it says nothing about whether the reused response was actually correct for the new query. Periodically sample served-from-cache responses and check them against what a fresh call would have returned. A high hit rate built on silently wrong reuses is a worse outcome than no caching at all, because it fails quietly and erodes trust in the feature without ever showing up as an error.