Issue #96

Test Your Output Parser Against the Model's Worst Day, Not Its Best

sent to 25 subscribers

Most output parsers are written while looking at three or four good responses. The JSON is clean, the fields are where you expect them, and everything works. Then the feature ships, and the parser meets the model on its worst day.

What does a worst day look like? A response wrapped in a markdown code fence. A friendly sentence before the JSON, or after it. A trailing comma. A field that is sometimes a string and sometimes a list. An output cut off mid-object because it hit the length limit. An empty response. A refusal, in plain prose, where you expected structure.

None of these are exotic. Over enough traffic, every one of them will show up, and your parser's behavior in those moments is part of your product whether you designed it or not.

Here is a habit that costs an afternoon and pays back for a long time. Take your parser and write down the ugly inputs as a fixture file: real malformed responses collected from logs, plus ones you invent deliberately. For each, decide in advance what the correct behavior is. Usually the answer is one of three things: recover (strip the fence and carry on), retry (ask again, once), or fail loudly with a specific error type that downstream code can handle.

The third option deserves emphasis. A parser that returns an empty object or a default value on failure is the most dangerous design, because a bad response then looks like a valid, empty one, and the bug surfaces three systems later as missing data.

Also keep the fixture file growing. Every time production produces a response that surprises you, add it. Over a few months it becomes a small regression suite of exactly the situations your model really generates, which is more valuable than any synthetic set.

Your parser will be judged by its behavior on the responses you did not expect. Test those first.

← Back to archive