Many teams postpone evaluations because they imagine a platform: dashboards, judge models, dataset management. You can get most of the value from something much smaller. This walkthrough builds a harness with three parts: a case file format, a runner, and a results format designed to be compared across commits.

It follows the principles in Anthropic's guide to developing tests, then adds the engineering details that make a harness usable day to day.

Principles to start from

The guide gives three design principles for evaluations:

  1. Be task-specific. Build evals that mirror the real distribution of your task, and include edge cases such as irrelevant or missing input, very long input, and ambiguous cases where even humans would struggle to agree.
  2. Automate when possible. Structure cases so they can be graded automatically: multiple choice, exact or string match, code-graded checks, or LLM-graded checks.
  3. Prioritize volume over quality. More cases with slightly lower-signal automated grading is better than fewer cases graded carefully by hand.

The third one is the permission slip to start small and cheap.

Part 1: Cases as plain files

Store each case as one line of JSONL, so diffs stay readable and cases are reviewed like code:

{"id": "refund-001", "input": "I was charged twice", "checks": [{"type": "contains", "value": "refund"}, {"type": "max_words", "value": 120}]}

Rules that pay off later:

  • Stable IDs. Results are compared by ID across runs, so never reuse or renumber them.
  • Checks live with the case. Each case declares what "good" means for it, starting with cheap deterministic checks: a substring present or absent, valid JSON, a schema match, a length bound.
  • Tags. Use a tags array (edge, regression, adversarial) so you can run subsets.

Part 2: A runner that stays boring

The runner loops over cases, calls your real application entry point (not a reimplementation of the prompt), applies the checks, and writes one result row per case. Details that separate a useful runner from a misleading one:

  • Repeat runs. Run each case N times (3 is a reasonable start) when output isn't deterministic, and record how many passed instead of a single pass or fail.
  • Record the inputs to the run: prompt version, model identifier and date. A result you can't attribute is a result you can't trust.
  • Catch exceptions per case and record them as failures. A crashed case must never silently disappear from the total.
  • Cap concurrency so the evaluation doesn't hit the same rate limits your production traffic relies on.

Part 3: Results you can diff

Write results sorted by case ID, one row per case, with the pass count, the run count and the names of the failing checks. Commit the file or store it per run. A prompt change then produces a reviewable artifact: "this change takes refund-001 from 3/3 to 1/3 and fixes shipping-014." That persuades reviewers far more than "accuracy went from 82% to 84%."

Add a small script that compares two result files and prints only the cases whose pass counts changed. That one script builds more team confidence than any aggregate score.

When to add a model as grader

Anthropic's guide covers several grading methods: exact match for categorical answers, embedding-similarity checks for consistency, ROUGE-style overlap for summaries, and LLM-based grading for subjective qualities such as tone, empathy or professionalism, either on a 1–5 scale or as a yes/no classification. Its advice for model-graded checks is worth following:

  • Use a different model to grade than the one that generated the output.
  • Define the criteria clearly in the grading prompt: state the scale or classes, give examples of each level, and ask for structured output, such as "output only the number."
  • Quantify vague goals. "Safe outputs" is not testable. "Under 0.1% of 10,000 outputs flagged by the content filter" is.

My rule: add a model grader only after a deterministic check has proven insufficient for a specific failure you have actually seen. Each grader is something you then have to maintain and trust, and it needs its own validation against human judgment on a sample.

Keep the set honest

An eval set that never changes gets overfit by iteration alone. Keep a final set you rarely touch, add new cases from real failures, and don't copy failing cases into the prompt as examples, or you'll be measuring memory. See Why a Held-Out Test Set Quietly Stops Being Held Out for the failure modes. Rerun the whole set whenever you change the model, as covered in Silent Model Upgrades Are a Production Risk.

The takeaway

The goal of the first harness isn't coverage. It's a habit: every prompt change ships with a before-and-after diff of named cases. Once that exists, heavier tooling has something real to build on, and you'll know which gaps it needs to fill.

Sources

Developer ToolsSoftware Engineering