If you serve a large language model yourself, the way you group requests decides how much work each GPU does per hour, often more than the choice of GPU. The technique that matters most is called continuous batching, and it is worth understanding even if you only ever use a framework that does it for you.

The problem with static batching

LLM generation is autoregressive: the model produces one token at a time, and each new token requires a full pass over the batch. The simplest way to serve many users is static batching. You collect a fixed group of requests, run them together, and start the next group only when every request in the current one has finished.

The flaw is that requests don't take the same number of steps. One user asks a yes/no question and gets a dozen tokens back; another asks for a long summary and needs several hundred. In a static batch the short request finishes early, but its slot stays occupied, doing no useful work, until the longest request is done. The more varied your traffic, the more GPU time is spent waiting.

What continuous batching changes

The idea was introduced in the 2022 OSDI paper on Orca, which proposed what it calls iteration-level scheduling: the scheduler decides what to run at the granularity of a single model iteration, not a whole request. After every iteration it can retire any sequence that has finished and admit a waiting request into the freed slot immediately. The batch never has to drain.

That sounds simple, but it only works because the paper also introduced selective batching, which applies batching to only a subset of the model's operations. Different sequences in the same batch are at different positions and lengths, so not every operation can be batched uniformly. Orca reported a 36.9× throughput improvement over NVIDIA FasterTransformer at the same level of latency, on a GPT-3 175B model. That is the paper's own benchmark on its own setup, so treat it as evidence of the direction and scale, not a number to expect on your workload.

"Continuous batching" and "in-flight batching" are the names the industry settled on for this approach.

Memory is the second half of the story

Admitting requests mid-flight creates a memory problem. Every active sequence keeps a KV cache, the stored attention keys and values for the tokens it has already processed, and each sequence's cache grows by one token per step. If you reserve a contiguous block sized for the maximum possible length, you waste most of it on short sequences, and the free space fragments as requests come and go.

The vLLM paper (SOSP 2023) attacked exactly this with PagedAttention, an attention algorithm inspired by operating-system virtual memory and paging. Its abstract claims near-zero waste in KV cache memory and flexible sharing of the cache within and across requests. Less wasted memory means more concurrent sequences fit on the GPU, which means bigger effective batches.

A common mix-up is worth correcting. You will often see "2–4×" quoted as the gain from continuous batching. That figure is from the vLLM paper, which reports a 2–4× throughput improvement over state-of-the-art systems such as FasterTransformer and Orca at the same latency. Orca already did iteration-level scheduling, so that gain comes mostly from better memory management on top of continuous batching, not from continuous batching alone. The two ideas compound.

What this means in practice

My own read, based on the papers rather than a measurement:

  • If you call a hosted API, you are already getting this. The provider's serving stack handles batching, and you can't change it. What you can change is how much you send per request and how you structure prompts.
  • If you self-host, don't write your own scheduler. Use a serving framework that implements continuous batching and paged KV-cache management. Rolling your own is how teams end up with a fragmented cache and a stalled GPU.
  • Benchmark with realistic traffic. The gains depend on how variable your request lengths are. A test with identical prompts and identical output lengths hides the very problem continuous batching solves, and will make every configuration look the same.
  • Watch the trade-off between throughput and latency. Packing more sequences onto a GPU raises total throughput, but each individual request can slow down. Set a latency target first, then see how much throughput you can get inside it.

The short version

Static batching wastes GPU time on whichever request is slowest. Continuous batching removes that wait by scheduling per iteration, and paged KV-cache management lets you keep far more sequences in flight. If you self-host an LLM and your stack still batches statically, this is usually the highest-leverage change available before you pay for more hardware.

Sources