Temperature Zero Is Not Determinism: What You Can and Can't Reproduce in LLM Calls

A common first move when an LLM feature behaves inconsistently is to set temperature to 0 and expect repeatable output. It usually helps. It rarely gives you the guarantee people assume, and building tests or caches on that assumption leads to confusing intermittent failures.

Where variation survives

Even with greedy decoding, several things can change the output between two identical-looking requests:

• Near-ties between tokens. When two candidates have almost the same probability, tiny numeric differences decide which wins, and one early token change can reroute the whole response.
• Numeric nondeterminism in serving. Parallel floating-point math isn't always evaluated in the same order, and how your request gets batched with others can shift results slightly. This is a property of the serving stack, not something you control.
• Model updates. If you call an alias rather than a pinned version, the model behind it can change. Identical input, different weights.
• Parameters that aren't available. Some models restrict or ignore sampling settings, and a seed parameter, where offered, is typically documented as best-effort. Read your provider's docs rather than assuming.
• Everything around the call. Retrieval results that reorder, timestamps in the prompt, and tool outputs that vary all change the input, which no sampling setting can fix.

Design for "mostly stable," not "identical"

Once you accept that exact repeatability isn't available, the engineering choices get clearer.

1. Test properties, not strings. Assert that the output parses, contains the required fields, cites a real source, or stays within a length. Exact-match assertions on free text will flake.
2. Record and replay for the deterministic layers. For tests of your own code, store a captured model response and replay it. That tests your parsing and logic without the model in the loop.
3. Measure variance directly. Run each eval case several times and report the spread. If a case flips between pass and fail across runs, that is information about the prompt's fragility, and a single run would hide it.
4. Cache by input hash when you need identical answers. If users or downstream systems require the same response for the same input, store the first response keyed by a hash of the full request. That gives you true stability, which sampling settings can't.
5. Pin model versions where the provider allows it, and log the version with each response so a behavior change can be traced to a cause.

The practical rule

Use temperature 0 to reduce noise, not to eliminate it. Where your system genuinely needs the same answer twice, get that from a cache or a stored result. Where it only needs answers of consistent quality, get that from evals that tolerate small variation and flag large variation. Knowing which of the two you need is most of the work.