Issue #68

Your Prompt Is Quietly Coupled to the Model You Tuned It On

sent to 25 subscribers

Spend enough time iterating on a prompt against one model and you'll start writing to that model's specific habits without noticing you're doing it: the exact phrasing that reliably gets it to use the right tone, the particular way you have to ask for JSON before it stops adding a preamble, the workaround for the one instruction it tends to ignore unless you repeat it. None of that is wrong, exactly — it's just normal iteration. But it creates a dependency nobody wrote down anywhere.

The dependency shows up the day you have a real reason to switch: a price change, a new model that's better at the actual task, an outage that pushes you onto a fallback provider. The prompt that scored well in your eval suite against the original model quietly underperforms against the new one, not because the new model is worse, but because your prompt was never really testing the task — it was testing the task as filtered through one model's idiosyncrasies, and you tuned until that filter stopped mattering to you.

The fix isn't avoiding model-specific tuning; some of it is unavoidable and worth doing. It's noticing when you're doing it, and periodically checking your eval suite against at least one other model even if you have no near-term plan to switch. If the score gap is small, your prompt is describing the task. If the gap is large, your prompt is describing the model, and that's worth knowing before the day you don't get to choose when you find out.

← Back to archive