Streaming LLM Responses Without Lying to Your Users: Partial Output, Cancellation, and Honest Endings

Streaming makes an LLM feature feel fast, but it quietly changes the contract between your backend and your UI. A non-streaming call either returns a complete answer or fails. A stream can do something in between: deliver half an answer and then stop. Most streaming bugs come from code that was written as if that middle state did not exist.

Treat the stream as a state machine, not a firehose

A stream has a beginning, many chunks, and an ending that can be clean or not. Model that explicitly. A useful minimal set of states is waiting (request sent, nothing received), receiving, complete, interrupted and cancelled. Each state should map to something the user sees and something your code does.

• `complete` requires a positive signal: the provider's final event or stop reason. A socket that simply closes is not completion. If you treat "the connection ended" as "the answer is done", a dropped connection will present a truncated answer as a finished one.
• `interrupted` means you received some text and then lost the stream. Decide up front whether you keep the partial text, label it as incomplete, and offer a retry. Silently replacing it, or silently leaving it, are both worse than saying so.
• `cancelled` is the user pressing stop or navigating away. It is not an error and should not page anyone.

Also keep the partial text and the final text as separate concerns. Anything you persist, index, or feed to downstream code (parsers, tool executors, citations checkers) should be written only once the state is complete. Rendering partial text is a display feature; acting on it is a bug.

Cancellation has to reach the provider

When the user closes the tab, the cheap mistake is to stop reading on the client and carry on. The server keeps pulling tokens you will never show, and you pay for all of them. Propagate cancellation through every hop: browser to your API, your API to the provider request. Most HTTP clients support aborting a request; wire the client disconnect event to that abort, and test it by measuring billed output tokens for a cancelled request against a completed one.

Cancellation also needs a decision about side effects. If the model already emitted a tool call before the user cancelled, did it run? Make that answer deliberate. A reasonable default is that tool calls only execute from a complete message, so cancelling mid-stream guarantees nothing happened.

Handle structured output separately

If you stream JSON, a partial document is not valid JSON, and a tolerant "repair the closing brackets" parser will happily hand you objects with missing fields that look fine. Either stream plain text for display and request a structured result at the end, or stream structure but validate the full schema before using any of it.

A short checklist

1. Success is a final event, never a closed connection.
2. Persist and act only on complete output.
3. Show interrupted output as interrupted, with a retry.
4. Abort the upstream request when the client leaves, and verify the token count drops.
5. Validate structured output after the stream ends.

The fastest-feeling stream is worth nothing if users cannot tell a finished answer from a cut-off one. Make the ending explicit and everything else gets easier to reason about.