Thumbs-up and thumbs-down buttons are sparse: most users never click them. But anyone using an AI draft feature, such as email replies, summaries, or generated descriptions, produces a much richer signal on every use. They edit the draft, or they do not. How much they change is one of the most honest quality measures you can get for free.

What edit behavior tells you

Consider the outcomes for a single generated draft:

  • Sent as is. The draft was good enough, or the user did not care enough to check. Both are worth knowing.
  • Lightly edited. Typically a tone fix, a name, or a detail. The draft was close, and the edits show which details the model cannot know.
  • Heavily rewritten. The draft missed the mark. The user paid the cost of reading it and then wrote it themselves, which can be worse than having no feature.
  • Discarded and regenerated. A clear rejection, and the regeneration pattern shows whether users give up after one retry or three.

Track these as a distribution, not an average. A feature where 60% of drafts go out untouched and 30% are rewritten completely has a very different problem than one where everything is edited a little.

How to measure it without fooling yourself

Pick a similarity measure that matches the content. Character-level edit distance works for short text. For longer text, compare at the sentence or paragraph level, because reordering a paragraph is a small edit in meaning but a large one in characters. Log the measure alongside the draft, the final text, and the prompt version, with appropriate privacy controls on stored text.

Be careful with three traps:

  1. Unedited does not mean correct. Users skim and approve. Pair the unedited rate with a spot check where a reviewer reads a sample of untouched drafts for errors.
  2. Edits have different meanings. Fixing a typo and correcting a factual error both count as small edits. Classify a sample of edits by hand into categories like tone, factual, and structure, then see which dominate.
  3. Selection effects. Users who rewrite heavily may stop opening the feature. Watch usage over time next to edit rates, or your numbers will improve just because unhappy users left.

Turn it into improvements

The most practical use is to read the heavily rewritten cases. Each one is a ready-made eval example: the input, the bad draft, and what the user wrote instead. A few dozen of these, grouped by cause, will usually point at one or two fixable prompt or context problems. Add them to your regression set so that the next prompt change is tested against what users actually did, not what you imagined they would.

An edit-rate metric will not tell you the model is good. It tells you where users are doing work the model should have done, and that is exactly where to look next.

AI ToolsSoftware Engineering