Most teams building an LLM feature eventually add a thumbs-up and thumbs-down button. Most of them then watch the counts on a dashboard and do nothing else with it. The ratings are far more valuable as raw material for your eval set, but only if you handle them carefully.

What a rating actually tells you

A thumbs-down says something went wrong, not what. It could be a wrong answer, a correct answer in the wrong tone, a slow response, or a user who mis-clicked. A thumbs-up is weaker still: users often approve answers they could not verify. So treat feedback as a pointer to cases worth examining, not as a label you can trust.

That framing changes the workflow. The goal is not to compute a satisfaction percentage. The goal is to find the specific requests where your system is weakest and turn them into regression tests.

A pipeline that works

  1. Capture context with the click. Store the request, the full response, the prompt version, retrieved documents or tool results, and the model identifier. A rating without the exact inputs is unreproducible and therefore useless.
  2. Prioritize negatives, plus a random slice of positives. Negative ratings find failures; a sample of positives lets you check whether your "good" is actually good, and keeps your set from becoming a catalog of only the worst days.
  3. Triage by hand, briefly. For each candidate, decide: real failure, unclear, or noise. Write one sentence on why. This reason is the most valuable field you will produce, because it tells you which failure category to fix.
  4. Write an expectation, not just an answer. For confirmed failures, record what a good response must do (cite the policy, ask a clarifying question, refuse politely) rather than one exact string. That keeps the test valid when wording changes.
  5. Scrub before you save. Remove or replace personal data before a user's text enters a shared test set, and check that your terms and retention rules allow reuse at all.

Watch for the biases

Feedback comes from the users who bother to click, who are not a random sample. Heavy users and frustrated users are overrepresented. Tag each case with the user segment and the request type so you can see which areas the feedback covers and which it silently ignores. Also be careful about mixing these cases into data you tune prompts against: hold some out so you can still measure honestly.

Close the loop

The payoff arrives when a fix ships and the old complaint becomes a permanent test that fails if the problem returns. Start small: pull the ten most recent negatively-rated conversations this week, write a reason and an expectation for each, and add them to your suite. Ten well-annotated cases from real use will teach you more than a hundred invented ones.

Large Language ModelsSoftware Engineering