Client-Side Rate Limiting for LLM APIs: Token Buckets, Backpressure, and Fair Sharing

Most teams meet rate limits the same way: a batch job or a traffic spike produces a wall of 429 responses, someone adds retries with backoff, and the problem seems solved. It is not. Retries convert a hard failure into a latency spike, and when many callers retry together they keep the limit saturated for everyone. A better approach is to limit yourself on the client side, before the provider has to do it for you.

Limit on the units the provider actually counts

LLM providers typically meter at least two things: requests per minute and tokens per minute (often input and output separately). A limiter that counts only requests will happily send ten huge prompts that blow through the token budget. Build one token bucket per dimension and require a call to acquire from all of them before it goes out.

Two practical details matter here:

• Reserve, then reconcile. Output length is unknown up front, so reserve against your max_tokens setting (or a realistic estimate), then refund the difference when the response reports actual usage. Reserving too little means overshooting; never refunding means your limiter is permanently more conservative than the real limit.
• Leave headroom. Run at something like 80-90% of the published limit. Other processes, other environments sharing a key, and clock skew between your window and the provider's all eat into the margin.

Add backpressure, not just a queue

A limiter that silently queues work hides overload. If requests wait thirty seconds for tokens, the user-facing caller has probably already timed out, and you are spending budget on answers nobody will read. Give every call a deadline. If the limiter cannot grant capacity before the deadline, fail fast with a clear "busy" error the caller can handle: degrade to a cheaper path, show a retry message, or shed the request.

Separate interactive traffic from batch traffic with distinct limiters, or one priority queue in front of a shared one. Without that, a nightly backfill will starve the chat feature that has a human waiting on it. Reserving a floor of capacity for interactive work, and letting batch use whatever is left, is a simple scheme that works well.

Share fairly across callers

If several services or tenants share one API key, the limit is a shared resource. Per-tenant buckets, sized as a fraction of the total, stop one noisy caller from consuming everything. If you run multiple processes, the limiter has to be shared too (a small Redis-backed counter is enough) or each instance will assume it owns the whole budget.

Treat 429 as a signal to adapt

Even a good limiter will occasionally be wrong. When a 429 arrives, honor the retry-after header, and also shrink your local rate briefly so you stop pushing against the ceiling. Recover gradually, in the spirit of additive-increase, multiplicative-decrease.

The closing point: rate limits are a constraint of the system you are building on, the same as memory or disk. Measure how close you run to them, alert on sustained 429 rates rather than individual ones, and design the overload behavior deliberately. If you decide what gets dropped ahead of time, an overload costs you some throughput. If you leave it undecided, the provider picks for you, and its choice is usually the worst one for your users.