A static few-shot prompt contains the same three or four examples for every request. That is a bet that those examples represent everything users will send. It works until inputs get diverse, and then the examples that help one request mislead another.

Dynamic selection replaces the bet with a lookup: keep a bank of labeled examples, and for each request choose the few that are most relevant.

The basic mechanism

  1. Keep an example bank: input and ideal output pairs, with metadata such as category, difficulty, and source.
  2. At request time, score bank entries against the incoming input. Embedding similarity is the common choice, but metadata filters or keyword matching are often enough and easier to debug.
  3. Insert the top few into the prompt in a consistent format.

The payoff is largest when inputs fall into distinct clusters, such as different document types or customer segments, and a single fixed set can't cover all of them.

Where it goes wrong

  • Near-duplicate selection. Pure similarity returns five almost identical examples, which teaches the model one pattern and wastes tokens. Diversify the picks, for example by penalizing candidates similar to ones already chosen.
  • Copying the example. When the closest example is *very* close, the model may copy its answer rather than reason about the new input. Watch for outputs that mirror a retrieved example's specifics.
  • Contaminated evals. If your bank contains items from your test set, retrieval hands the model the answer. Keep the bank and the eval set strictly separate, and check overlap programmatically.
  • Unreviewable behavior. The prompt now differs per request, so a bad output is hard to reproduce. Log which example IDs were selected for each call so you can replay it.
  • Bank rot. Examples encode past policy. When requirements change, stale entries quietly pull outputs back to the old behavior, so give them owners and review dates.

How to know it's worth it

Compare against a good static baseline on the same eval set, and break results down by input cluster. Dynamic selection often shows its gains on the hard clusters while changing little elsewhere. If the overall number barely moves, a better-written static prompt may be the cheaper fix.

The closing point: dynamic few-shot turns part of your prompt into a retrieval system, so treat the example bank like any other data source, with versioning, logging, and tests. The gain is real when it is measured, and a liability when the selection logic is left unobserved.

Developer ToolsLarge Language Models