Most teams wire every LLM call the same way: a request comes in, the code calls the model, and the user waits. That's right for a chat window. It's the wrong default for the growing share of LLM work that isn't chat: classification, enrichment, tagging, summarization, extraction. That work runs on a schedule or a queue, with nobody watching a spinner.

The tell is usually a cost or rate-limit problem blamed on the model. A nightly job that classifies 200,000 support tickets through the same real-time endpoint your chat feature uses pays the same price per token and competes for the same limits, even though nothing about it is time-sensitive. Batch APIs exist for exactly this. The specifics below come from Anthropic's Message Batches documentation; other providers offer similar products with their own terms.

How a batch API works

Per the docs, you submit a set of requests as one batch. The system processes each request independently and asynchronously, and you poll for the batch's status and retrieve the results when processing has ended. The headline numbers:

  • Cost: 50% lower than standard calls.
  • Speed: most batches finish in less than an hour. The hard limit is 24 hours: a batch expires if it hasn't completed by then.
  • Size: up to 100,000 requests or 256 MB per batch, whichever you hit first.
  • Results: available for 29 days after the batch is created, after which the batch is still visible but its results can't be downloaded.

You give every request a custom_id (1–64 characters: letters, digits, hyphens and underscores) so you can match results to inputs. That matters because results aren't guaranteed to come back in the order you sent them.

Four outcomes per request

Each request in a batch ends in one of four states:

| Result | Meaning | Billed? | |---|---|---| | succeeded | The request worked, and the message result is included. | Yes | | errored | The request failed, for example because it was invalid or due to a server error. | No | | canceled | You canceled the batch before this request was sent to the model. | No | | expired | The 24-hour window ended before this request was sent. | No |

So your job code needs a loop: send, poll, split the results by outcome, and resubmit the errored and expired ones. Retry logic moves from the request level to the job level.

What really needs a synchronous call

List every LLM call site and write down its real latency requirement, not the assumed one.

  • Synchronous: a chat reply, an agent step that blocks a UI, anything where seconds matter.
  • Batchable: nightly re-embedding of a document store, a sentiment pass over yesterday's reviews, extraction of fields from a backlog of PDFs, evaluation runs over a test set, bulk reclassification for weekly reporting.

Most of the batchable work inherited the synchronous pattern because the first LLM call in the codebase was written that way and every later one copied it.

Split the pipeline, don't convert it

Don't treat this as all-or-nothing. A ticket-triage system might classify synchronously the tickets a human is looking at, while the historical reclassification behind a weekly report runs as an overnight batch. Write the extraction logic once, behind an interface that doesn't care whether the answer comes back in two seconds or twenty minutes, and let the caller choose.

Cautions

  • Plan for partial failure. Expect some expired and errored requests and make resubmission routine.
  • Don't batch what a user is waiting for. "Usually under an hour" is not a latency guarantee.
  • Don't count on caching savings. Anthropic's docs say you can use prompt caching with batches, but because requests are processed concurrently and in any order, cache hits are best-effort. Budget for the batch discount, and treat caching as a bonus.
  • Check data-retention terms. Anthropic's batch page points to its zero-data-retention documentation for how that applies; check the equivalent for your provider and your obligations.

The audit worth doing this week

Mark which of your LLM call sites actually block a human, and move the rest to a queue. In many codebases that's a third or more of call volume, and it's often the part generating the most cost and the most rate-limit complaints.

Sources

Developer ToolsLarge Language ModelsSoftware Engineering