When something goes wrong in an AI feature, the instinct is often to reach for interpretability: to ask why the model produced this particular output, as if the answer lives somewhere in its internals and needs to be excavated. For the overwhelming majority of production incidents, that instinct points in the wrong direction — and worse, it delays the fix.
Most of what actually breaks an AI feature is not a mystery about model reasoning. It is a question you could answer with a good log line: what exact text was in the prompt when this ran, which tool calls fired and what they returned, which documents retrieval pulled back and in what order, which prompt version and model version were live at that timestamp, and what — if anything — changed since the last run that worked. That is closer to observability than interpretability, and it is a problem engineering teams already know how to solve.
The reason this matters is that interpretability and observability call for different investments. Interpretability research is genuinely hard and largely unsettled; waiting on it to debug today's incident is waiting on the wrong thing. Observability is a solved problem in every other part of a production system, and there is no good reason an AI feature should be the one component where you cannot reconstruct exactly what happened.
Before spending effort on exotic explainability tooling, make sure you can already answer the boring question for any given output: what exactly did the model see, and has that changed since yesterday. Most explainability problems, in practice, turn out to be logging problems wearing a more interesting name.