Teams often pick a model first and discover its latency second. By then the product flow is built around it, and "make it faster" becomes a vague request with no finish line.
Flip the order. Before choosing a model or an architecture, write down a latency budget for the feature, in terms a user would recognize:
- How long will someone wait before they assume it's broken? For an interactive action, that is often a couple of seconds. For a background task, it may be minutes. The answer changes everything about the design.
- What is the time to first useful output, versus time to complete? Streaming can make a long generation feel responsive, but only if the first tokens are actually useful and not a preamble.
- What is the worst case you will tolerate? Averages hide the slow tail. Set the budget at a high percentile, such as p95, because that is what your most frustrated users experience.
Then spend the budget deliberately. A request might include retrieval, one or more model calls, tool calls, and post-processing. Give each stage a share, and decide up front what happens when a stage overruns: return a partial result, fall back to a cheaper path, or tell the user honestly that it is taking longer.
A few levers are worth knowing about, in rough order of effort: shorten the prompt and the expected output length, reorder the prompt for cache hits, run independent calls in parallel, use a smaller model for the easy cases and reserve the larger one for hard ones, and cache results for repeated inputs.
The point of the budget is not to hit a number for its own sake. It turns "is this good enough?" from an argument into a measurement, and it tells you early when a candidate model cannot work for this feature, before you have spent weeks integrating it.
This week: pick one LLM feature you own, write its latency budget in one sentence, and check whether your dashboards can even show you the p95. If they cannot, that is your first task.