A Framework for Deciding Where an LLM Call Belongs in Your Request Path

Teams spend enormous effort tuning prompts and comparatively little effort deciding where in the request path an LLM call should actually live. That placement decision — synchronous in the request-response cycle, streamed to the client, or pushed to a background job — has more effect on perceived quality, cost, and reliability than most prompt changes do, and it's worth a deliberate framework instead of defaulting to whatever was easiest to prototype.

Three questions before you pick a placement

Start with how long the user is willing to wait attentively. If the answer is "a couple of seconds, in the middle of a task," synchronous or streaming is the only real option — background processing will just make the feature feel broken, not asynchronous. If the answer is "not watching, checking back later" — a report, a batch classification, a summary of yesterday's activity — a background job is almost always the better choice, because it can retry, use a slower and cheaper model, and never blocks a page load on a flaky API.

Second, ask what happens on failure. A synchronous call that fails has to fail visibly and immediately, in a UI that can show a retry button. A background job that fails can retry silently, escalate to a fallback model, or notify only if it fails twice — options a synchronous path doesn't have room for. If your current design streams a call that regularly needs three retries to succeed, that's a sign the call belongs in a job queue, not the request path.

Third, look at whether the output needs to inform something the user does immediately after. If the next UI state depends on the LLM's output, it has to be in the critical path in some form. If it doesn't — if the output enriches a record the user won't look at for hours — moving it out of the critical path is close to free.

The cost of getting it wrong in either direction

Putting a slow, retry-prone call synchronously in the request path teaches users that the feature is slow and flaky, even when the model itself is fine — the failure is architectural, not a model quality problem, but users can't tell the difference. Going the other way — pushing something to a background job that the user actually needed right away — creates a worse failure: a UI that looks broken or stale because the result the user is staring at hasn't arrived yet.

The fix isn't a universal rule; it's writing the three questions above into your design review checklist for any new LLM-backed feature, so placement gets the same deliberate attention prompt wording already gets.