A common pattern in retrieval-backed features is to ask the model to cite its sources. That is a good instinct, but a citation is only a claim. Models can attach a plausible-looking reference to a sentence the source never supported, or paraphrase a quote so loosely that it no longer matches. The good news is that a surprising share of this can be caught with no second model call at all, just deterministic checks.

Ask for quotes, not just references

The first step is changing what you request. A bare "[3]" tells you which document the model says it used, but gives you nothing to verify. Instead, have the model return a short verbatim quote alongside each claim, in a structured field:

  • the claim it is making,
  • the source identifier,
  • the exact span copied from that source.

Now verification becomes a lookup. Normalize whitespace and case on both sides, then check that the quoted span appears in the retrieved document text. If it does not appear, the citation is unverified. This is plain substring matching, which means it is fast, free, and fully reproducible in a test.

What this does and does not prove

Be precise about the guarantee. A quote that exists in the source proves the model did not invent the *text*. It does not prove the quote *supports the claim*. A model can faithfully copy a real sentence and still draw the wrong conclusion from it, or quote a passage that is about something adjacent. Treat mechanical checking as a floor, not a ceiling.

You can raise the floor cheaply:

  • Require the source to be one that was actually retrieved for this request. A citation to a document that was never in the context is a fabrication, and you can detect it by checking IDs against the retrieved set.
  • Reject very short quotes. A three-word span will match almost anywhere. Set a minimum length so matching means something.
  • Allow fuzzy matching carefully. Exact matching will flag harmless differences such as curly versus straight quotes. A small edit-distance tolerance is reasonable, but keep it tight so you do not wave through paraphrases.

Decide what failure does

Verification is only useful if something happens on failure. Reasonable options include dropping the unverified claim, regenerating once with the failure fed back, or showing the answer with the unsupported citation visibly marked. Which you pick depends on stakes: a legal or medical context should withhold, a casual search assistant can annotate.

Also record the verification rate as a metric. A sudden drop usually points at something concrete, such as a chunking change that splits sentences, a prompt edit, or a model swap, and it shows up long before users complain.

The closing point: verification does not require a clever judge model. Start with what you can check deterministically, be honest about what that check proves, and reserve model-based grading for the question string matching cannot answer, which is whether the quote supports the claim.

AI IntegrationLarge Language ModelsSoftware Engineering