When an LLM is asked for JSON, most responses parse and a few do not. The few are where production incidents live: a missing closing brace, a field of the wrong type, an enum value the model invented. The usual reflex is "retry and hope", and sometimes that works. A deliberate repair loop works much more often, and it fails in ways you can predict.

Validate first, then decide what kind of failure you have

Parsing is only the first gate. Treat validation as three separate checks, because each failure calls for a different response:

  1. Syntax: does the text parse as JSON at all? Failures here are usually truncation or stray prose around the payload.
  2. Schema: are required fields present, with the right types and allowed values?
  3. Semantics: do the values make sense together? A end_date before start_date is valid JSON and valid against the schema, and still wrong.

Log which check failed. Over a few weeks, the distribution tells you where to invest: mostly syntax errors point at output limits or formatting instructions, mostly schema errors point at an unclear schema, and semantic errors point at the task itself.

Repair with the error, not with the same request

A blind retry resends the same prompt and gets another sample from the same distribution. A repair attempt gives the model new information: its own previous output and the specific validator message.

A workable repair prompt contains:

  • the original instructions (or a reference to them),
  • the invalid output, quoted verbatim,
  • the exact errors, for example "priority": "urgent" is not one of ["low", "medium", "high"],
  • an instruction to return only the corrected object, changing nothing that was already valid.

That last clause matters. Without it, models sometimes "fix" the error by quietly rewriting other fields, and you have traded a loud failure for a silent one.

Cheap repairs before model repairs

Some failures do not need a model call. Stripping a Markdown code fence, trimming text before the first {, or coercing "3" to 3 where the schema is unambiguous can be done in code. Keep these deterministic fixes small and logged. If you find yourself writing a dozen of them, the prompt is the thing to fix.

Bound the loop and fail honestly

A repair loop needs a hard ceiling. One or two attempts is typically enough; if the model cannot satisfy the schema after being shown the errors, a third try rarely helps and mostly adds latency and cost. When the budget is spent:

  • return a typed failure to the caller rather than a half-valid object,
  • keep the raw output and error list for debugging,
  • let the calling code choose a fallback, such as a default, a human review queue, or a user-visible error.

Also track the repair rate as a metric. A rate that creeps up after a prompt or model change is an early regression signal, often visible well before users complain.

The takeaway

Make structured-output handling a small pipeline: validate in layers, repair with the actual error message, cap the attempts, and fail with a clear result. The goal is not to make invalid output impossible. It is to make every invalid output either fixed correctly or surfaced loudly, never passed along quietly.

Large Language ModelsSoftware Engineering