Evaluating Summaries by Claims, Not Vibes: Separating Faithfulness From Coverage

Summarization is one of the most common LLM features and one of the hardest to evaluate. Asking "is this a good summary?" invites a vague answer from a human or a judge model, and the scores drift depending on who is grading and what mood they are in. A more reliable approach is to break a summary into checkable claims and ask two separate, narrower questions.

Two questions, not one

A good summary has two properties that fail independently:

• Faithfulness: every statement in the summary is supported by the source.
• Coverage: the important points in the source appear in the summary.

A summary can be perfectly faithful and still useless because it only mentions trivia. It can cover everything and still contain one invented number that ruins it. Folding both into a single 1-to-5 score hides which failure you have, so you cannot tell what to fix.

Faithfulness: split the summary into claims

Take the model's summary and split it into atomic claims, one fact per line. You can do this with a prompt, since extracting claims from text the model just wrote is a much easier task than judging quality. Then, for each claim, check it against the source with a narrow question: "Is this claim supported, contradicted, or not mentioned in the source text? Quote the supporting passage."

Requiring a quote matters. It forces the check to point at evidence, and you can verify mechanically that the quoted passage actually appears in the source. Your faithfulness score becomes the fraction of claims that are supported, and the unsupported claims form a concrete list you can read.

Coverage: start from a checklist of key points

For coverage, work in the opposite direction. For a small set of source documents, have a person write down the three to seven points a good summary must include. These are your reference points. Then check, for each point, whether the summary expresses it. This is slower to build, but it is a one-time cost per test document, and it turns a fuzzy judgment into a count.

You do not need hundreds of documents. Thirty to fifty varied sources, including a few awkward ones such as tables, contradictory passages, and very short inputs, will reveal most problems.

Practical notes

• Calibrate any model-based checker against a few dozen claims you labeled yourself before trusting it.
• Report faithfulness and coverage separately, and track the list of unsupported claims over time.
• Watch for summaries that score well on both by simply copying sentences from the source. If extractive copying is not what you want, add a length or paraphrase check.
• Treat numbers, names, and dates as a special category. These are the claims readers act on, so check them with exact matching against the source whenever possible.

The takeaway is that evaluation gets more trustworthy as the questions get smaller. "Is this claim in the source?" has an answer. "Is this a good summary?" has an opinion. Decompose until every check has an answer, and your summarization quality stops being a matter of taste.