Tracing a Multi-Step LLM Request: Spans, Not Log Lines

A single user request to an LLM feature rarely maps to a single model call. It might rewrite a query, retrieve documents, call two tools, draft an answer, and run a validator, with a retry or two along the way. When something goes wrong, a flat pile of log lines makes you reconstruct that story by hand. A trace gives you the story already assembled.

Model one request as a tree of spans

A span is a named, timed unit of work with a parent. The request is the root span; each stage is a child; each model call or tool call is a child of its stage. That structure answers questions log lines cannot:

• Where did the 9 seconds go? Look at the widest span, not at timestamps scattered across files.
• Which retry produced the answer the user saw? Retries appear as sibling spans under the same parent, in order.
• Did the tool call happen before or after the model asked for it? Parent-child order tells you.

Every span needs a small set of attributes you decide on up front: the stage name, the model identifier and prompt version, input and output token counts, the stop reason, and an outcome (ok, error, timeout, truncated). Consistent names matter more than rich ones, because you will be filtering on them at 2 a.m.

Decide what to store, and what not to

The temptation is to attach full prompts and completions to every span. That is useful for debugging and dangerous for privacy and storage cost. A workable compromise:

1. Always store metadata: sizes, timings, versions, outcomes, and hashes of the rendered prompt.
2. Store full text only for sampled requests, for requests that ended in an error, and for requests a user flagged.
3. Apply the same redaction you use anywhere else personal data flows, before the span is written, not after.

The hash is underrated. If two requests with the same prompt hash behave differently, you have ruled out the template and can look at the data or the model instead.

Propagate the trace through everything

A trace is only as good as its continuity. Pass the trace ID into every tool call, queue message, and background job the request spawns. If an agent hands work to another process and that process starts a fresh trace, you have two half-stories and no link between them. Put the ID in your outbound HTTP headers and in your job payloads, and make it a required argument in your internal LLM client wrapper so nobody can forget.

Use traces for more than firefighting

Once spans carry outcomes and token counts, aggregate them. Plot the share of requests that hit a retry, the p95 duration per stage, and the fraction of spans ending in truncated. These are cheap, always-on signals that a prompt or model change has shifted behavior, and they cost nothing extra to collect because you already emit the spans.

The concrete step to take this week: wrap your single model-call function so it emits one span with stage, model, prompt version, token counts, and outcome. Everything else in this article builds on that one chokepoint.