Streaming makes an LLM feature feel fast even when the total generation time hasn't changed. But "just turn on streaming" is not a UX design. Streaming introduces new states, new failure modes, and new places to be wrong in front of the user. This article covers how to think about time-to-first-token (TTFT) as a product metric and what to design around once tokens start arriving.

Measure two latencies, not one

Total response time and time-to-first-token answer different questions. TTFT is what the user perceives as "did anything happen?" Total time is what they perceive as "am I done waiting?" A feature with a 900 ms TTFT and a 12 second total can feel snappier than one with a 3 second non-streamed response, provided the text that arrives is readable as it arrives.

Track both as separate percentiles, and track them per request type. TTFT is dominated by things you can often control: prompt length, retrieval or tool calls that run before generation, queueing, and cold connections. Total time is dominated by output length. If your p95 TTFT is bad, shrinking the output won't help; trim the pre-generation work.

Decide what the user sees before the first token

If your pipeline does retrieval or tool calls before generation, the first several seconds are silent. Options, from weakest to strongest:

  • A generic spinner. Better than nothing, but it hides what is happening and doesn't tell the user whether to keep waiting.
  • Named stages. "Searching your documents", "Drafting an answer". These cost almost nothing and turn dead time into progress. They must reflect real pipeline stages, not a fake timer.
  • Early partial results. If retrieval finishes first, show the sources immediately while the answer is still being written.

Design for the stream that goes wrong

A streamed response can fail after the user has already read half of it. Plan for these cases explicitly:

  1. Mid-stream error. Keep the partial text visible, mark it clearly as incomplete, and offer retry. Silently replacing it with an error message throws away content the user may have been reading.
  2. Content that is retracted. If a moderation or validation step runs on the full output, you can't un-show tokens already displayed. Either validate before streaming, stream only from sources you are prepared to stand behind, or design the UI so that a retraction is visible and explained.
  3. Structured output. Rendering a half-received JSON object or a half-received Markdown table produces flicker and garbage. Render structure only when a unit is complete, and stream free text only in the parts of the UI designed for it.
  4. User interrupts. Provide a stop control and make sure stopping actually cancels upstream generation, otherwise you keep paying for tokens no one reads.

Don't let the stream hurt accessibility

Constantly updating text can overwhelm screen readers, and auto-scrolling can fight a user who scrolled up to reread something. Buffer updates into sentence-sized chunks for assistive technology, and stop auto-scrolling the moment the user scrolls manually.

The takeaway

Treat streaming as a state machine with explicit states: waiting, generating, complete, failed-partial, cancelled. If you can name what the interface shows in each of those states, you've designed the feature. If you can only describe the happy path, you've only turned on a flag.

AI IntegrationLarge Language ModelsSoftware Engineering