Most teams pick an LLM observability tool the way they once picked their first APM tool: try the one a teammate mentioned, glance at the dashboard, move on. That works until the first confusing production incident, when you find the tool captured the wrong things. A better approach is to evaluate tooling against the failures you'll actually have, not the features that demo well.

This article is a buyer's checklist. It's largely my own guidance from how LLM features fail in production, anchored where possible to what the vendors' own APIs and the open standards say.

Start from the standard, not the vendor

Before comparing products, look at the open standard they should be able to speak. OpenTelemetry maintains GenAI semantic conventions in a dedicated repository, which describes them as covering spans, metrics and events for GenAI clients, MCP (the Model Context Protocol) and provider-specific conventions. I could not find a stability status stated on the repository's front page, so check the current state yourself before you depend on it.

The practical questions for any vendor: can it ingest OpenTelemetry data, can you export your traces, and does it understand MCP tool calls? A tool that traps your data in a proprietary format raises the cost of switching later.

Trace completeness beats pretty dashboards

The most valuable property is whether the tool captures the entire call chain for one user request, linked by a single ID: every prompt sent, every tool call and its arguments, every retry, every fallback, and the final output. A tool that records only the top-level prompt and completion looks fine in a demo and is nearly useless the day a multi-step agent goes wrong three tool calls deep.

Test it on your messiest real workflow, the one with retries, branching and a fallback model, and check that you can reconstruct exactly what happened from one link.

Per-step cost and latency

"This request cost $0.04 and took 2.1 seconds" tells you nothing when it touched five prompts. You want per-step attribution: "Step 3, the reranking call, is 60% of the latency and 80% of the cost." Provider responses already give you the raw material. For example, Anthropic's API reports input tokens split into cache reads, cache writes and uncached tokens, which is what you need to see whether prompt caching is actually working. Check that the tool records those fields and doesn't flatten them into one total.

Record why each call ended

Make sure traces keep the stop reason alongside the text. A response cut off at the token limit looks like a success otherwise, and you'll want a metric on how often it happens. See Truncated Output Is a Silent Failure.

Version linkage

Each trace should record which exact prompt template, parameters and model identifier produced it. When behavior shifts, "what changed" should be a query, not an investigation. Model identifiers matter because models are retired and replaced on the provider's schedule, as covered in Silent Model Upgrades Are a Production Risk.

Replay and diffing

Dashboards tell you something changed. Replay, re-running a captured production request against a new prompt or model and seeing the difference, is what lets you fix it and confirm the fix. Ask vendors to demonstrate replay on a trace that includes a tool call, not only a single-turn completion. That is where many implementations fall apart.

Privacy and retention

Traces contain prompts and outputs, which often contain personal or confidential data. Check what the tool stores by default, whether you can redact fields or turn off content capture, where data is hosted, and how long it is kept. Decide this before the first production trace, not after.

Run a proper trial

  1. Send the vendor your worst production trace: a retry, a tool call and an unexpected fallback in one request.
  2. Ask them to explain what happened using only what the tool recorded.
  3. Check export and retention terms in writing.
  4. Estimate the cost at your real traffic volume, including trace storage.

If the tool can't explain your worst trace, no dashboard will make up the difference later.

Sources

AI IntegrationAI ToolsDeveloper Tools