Wrapping an LLM call in a retry loop is one of the first things teams do to harden an AI feature, and it's usually the right instinct. Transient timeouts, rate limits and malformed outputs are common enough that failing on the first attempt is worse than trying again. The problem isn't retrying. It's treating the retry as free, when it changes your cost, latency and correctness in ways a normal HTTP retry doesn't.

The foundation for thinking about this is Amazon's Builders' Library article "Timeouts, retries and backoff with jitter," which this piece draws on, adding what is specific to LLMs.

The general problem: retries add load

Amazon's article makes a blunt point: when failures are caused by overload, retries that add load can make things significantly worse. It calls retries "selfish," because each one asks the server for more resources at exactly the moment resources are short.

The article's most striking example is about stacked services. With five layers, each retrying three times, the load on the database at the bottom increases 243-fold, making it unlikely to ever recover. The recommendation is to retry at a single point in the stack. That applies directly to LLM features: if your SDK retries, your gateway retries and your app retries, one failure becomes many calls, and every one is billed.

The standard tools

The same article gives the usual remedies:

  • Capped exponential backoff. Wait longer after each failure, with an upper limit so the wait doesn't grow without end.
  • Jitter. Add randomness to the delay so clients don't all retry at the same instant.
  • A retry budget. Amazon describes using a token bucket: calls may retry while tokens remain, then fall back to a slow, fixed rate once the bucket is empty. That caps the extra load a failing dependency can attract.

What's different about an LLM call

These points are my own analysis of how LLM calls differ from ordinary API calls:

Cost doesn't reset. A retried database query costs about the same as the first. If an LLM call fails after generating most of a long response (a dropped connection mid-stream, a truncated JSON blob), you have paid for those output tokens and will pay again, with no partial credit. Tail latency makes it worse: the slowest calls are the ones most likely to time out and retry, so your worst-case latency and worst-case cost coincide.

A retry of the same request may fail the same way. If a response was cut off by the token limit, an identical retry truncates in the same place. Fix the limit instead. See Truncated Output Is a Silent Failure.

Retries and side effects don't mix by default. Amazon's rule is that APIs with side effects aren't safe to retry unless they provide idempotency. For agents that's the central issue: if the model calls a tool that sends a message or charges a card before the failure, a timeout doesn't tell you whether the action happened. Retrying blindly doesn't recover; it duplicates the action. Give every side-effecting tool an idempotency key before you retry the step that triggers it.

A policy, not a wrapper

  1. Decide what's retryable. A pure generation call with no side effects can be retried automatically. A call where the model can invoke tools needs idempotency first.
  2. Retry at one layer. Turn off the duplicates in SDKs and gateways, and keep one policy you can reason about.
  3. Back off with jitter, and cap attempts. Two or three tries is usually enough. Don't retry errors that won't change, such as an invalid request or an exhausted account balance.
  4. Put a budget on it. Limit retries as a share of traffic so a provider outage doesn't multiply your load and your bill.
  5. Measure it. Log attempt counts and the tokens spent on failed attempts, as part of your observability. A retry rate that creeps up is an early warning.
  6. Prefer batch for non-urgent work. Jobs with no human waiting are better handled with batch APIs, which handle per-request failure for you.

The takeaway

Retries are still the right default for LLM calls, since surfacing every transient hiccup to a user is worse. But "retry on failure" is a policy with cost, latency and correctness trade-offs attached, not a one-line wrapper you add once and forget.

Sources

AI IntegrationDeveloper ToolsSoftware Engineering