Teams reach for fine-tuning earlier than they need to, and they stop adding examples to a prompt too soon. Both mistakes come from treating the choice as a maturity ladder, where "real" teams fine-tune, instead of as an engineering trade-off with a fairly clear decision boundary.

The most useful public framework for the decision comes from OpenAI's guide to optimizing LLM accuracy. This article summarizes it and adds practical rules of thumb, which are marked as mine.

Diagnose the problem before choosing a tool

OpenAI's guide separates two kinds of problem and matches each to a different fix:

  • Context problems. The model lacks knowledge because it wasn't in its training set, is out of date, or is proprietary. The fix is context optimization: retrieval-augmented generation (RAG), which provides the knowledge at request time.
  • Behavior problems. The model gives inconsistent results, wrong formatting, the wrong tone, or doesn't follow reasoning consistently. The fix is LLM optimization: better prompting and, when that plateaus, fine-tuning.

Fine-tuning mostly teaches behavior, not facts. If the model doesn't know your product catalogue, training on a few examples is the wrong tool; retrieval is. If it knows enough but formats output inconsistently, no retrieval will fix that.

The guide gives a case where this matters. In an example of correcting Icelandic text, a fine-tuned model reached an 87 BLEU score, and adding RAG on top lowered the score to 83. The guide's moral is to use the right optimization for the right job, and not to stack methods by default.

The guide's sequence, in short:

  1. Start with prompt engineering and an evaluation set. This establishes a baseline and defines what "accurate" means for you.
  2. Diagnose failures as context problems or behavior problems.
  3. Apply RAG for context gaps or fine-tuning for consistency issues.
  4. Stack methods only when you have both kinds of problem.

Notice that the evaluation set comes first. Without one you can't tell whether any step helped.

Where few-shot prompting wins (my rules of thumb)

These are my own heuristics, not from the guide:

  • The rules change often. Swapping a prompt is cheap. Retraining isn't, and a fine-tuned model locks in whichever definition of "correct" it was trained on.
  • You have few labeled examples. If you have dozens, not thousands, there isn't enough to fine-tune well.
  • You need to explain why. A well-chosen example with a short rationale can teach a boundary that a raw label can't.
  • You serve several variants (tenants, schemas) from one system. Different prompts are easier to manage than several fine-tuned models.
  • You're still discovering what "good" looks like.

Where fine-tuning earns its cost (also my heuristics)

  • A large, stable set of labeled examples exists and the task definition has stopped moving.
  • Prompting has plateaued on a behavior problem, such as an output format the model won't hold or a hard-to-describe house style.
  • Your few-shot examples are bloating every request. At some point, paying tokens for ten examples on each call costs more than training once. Check whether prompt caching already makes the examples cheap before deciding.
  • A smaller fine-tuned model could cut latency or cost while matching the larger model's quality on your task.

A three-point check before fine-tuning

  1. The task definition hasn't changed in your last few review cycles.
  2. Few-shot prompting has demonstrably plateaued on your evaluation set, not just in someone's assumption.
  3. You can compare the fine-tuned model against the best prompted version you already have, not only against the base model.

The third point is the one teams skip. A fine-tuned model that is worse than a well-prompted base model is a real and common outcome, and it gets shipped because "fine-tuned" sounds better. Sort out what's missing from your context first. If you can't tick all three, the honest answer is one more evaluation-driven iteration on the prompt.

Sources

AI ToolsLarge Language Models