A request that touches an LLM rarely touches it once. A typical pipeline retrieves documents, rewrites the query, calls a model, validates the output, and sometimes calls a second model to repair it. Each of those calls usually has its own timeout, and each timeout is chosen in isolation. Add them up and you get a worst case nobody agreed to: five stages with a 30-second timeout each is a request that can hang for two and a half minutes while the user's browser gave up at 30 seconds.

The fix is old and unglamorous: propagate a deadline, not a timeout.

Timeouts are local, deadlines are global

A timeout answers "how long may this call take?" A deadline answers "by what wall-clock moment must the whole request be finished?" The second question is the one your user is actually asking.

The pattern is simple:

  1. At the edge of the system, compute deadline = now + total_budget.
  2. Pass the deadline (not a duration) to every stage, as a parameter or in request context.
  3. Before each stage starts, compute remaining = deadline - now. If it is below the minimum useful time for that stage, skip the stage or fail fast instead of starting work that cannot finish.
  4. Give each outbound call min(stage_cap, remaining) as its timeout.

Passing an absolute time rather than a duration matters because durations drift: every layer that forwards "30 seconds" forgets how much was already spent upstream.

Decide what to sacrifice, in order

A deadline is only useful if you have decided what gives when time is short. Write that down per stage:

  • Optional enrichment (reranking, a second retrieval pass): skip when remaining time is under a threshold.
  • Quality improvements (a self-check or repair call): skip and return the first answer, flagged as unverified if your product needs that distinction.
  • Essential stages (the main generation): never starve these. Reserve their share of the budget up front instead of letting earlier stages eat it.

Reserving is the part teams forget. If generation needs at least 12 seconds to be useful, then retrieval gets total_budget - 12 at most, regardless of how slow the vector store is feeling today.

Don't forget the cleanup

When a deadline passes, cancel the in-flight work, not just the wait. An abandoned LLM call that keeps streaming tokens still costs money and still occupies a concurrency slot, which makes the next request slower and the next deadline harder to meet. Make sure your HTTP client actually closes the connection on cancellation, and log which stage was running when the deadline hit.

That last log line is the payoff. Once every timeout is a deadline miss attributed to a named stage, "the AI feature is slow" turns into "the rerank stage consumed 40% of the budget on Tuesday afternoons," which is a problem you can actually fix.

The takeaway: choose one total budget from what the user will tolerate, spend it explicitly across stages, and decide in advance which stages are allowed to be skipped. Per-call timeouts are a safety net; the deadline is the design.

AI IntegrationSoftware Engineering