Most teams treat an eval set as an asset you build once and then reuse indefinitely. That works for a while, and then it quietly stops working.
An eval set is a snapshot of what you believed users would ask, and what a good answer looked like, on the day you wrote it. Both of those drift. Users find new ways to use the feature. The product adds capabilities the old cases never touch. Documentation changes, so a case whose "correct" answer came from last year's policy is now rewarding the wrong behavior. Meanwhile your prompts have been tuned against the same cases for months, so a rising score increasingly measures how well you fit those cases rather than how well you serve people.
The symptom is familiar: the dashboard is green, and support tickets describe failures nobody's eval caught.
Three habits that help
Date your cases. Record when each case was added and where it came from. When you can see that half of your suite predates the last two product changes, you know where to look.
Feed it from production. Every real failure that reaches you should become a case, with the original input and a corrected expected result. This is the cheapest source of cases that reflect what users actually do, and it keeps the suite honest about your weak spots.
Retire and review on a schedule. Once a quarter, sample some cases and ask: is the expected answer still right, is this scenario still possible, does it still distinguish a good system from a bad one? Cases that every version passes trivially carry no information, so remove or replace them. Keep a small frozen subset for tracking long-term trends, and label it as such.
None of this requires new tooling, just a column for the date and a calendar reminder. The goal isn't a bigger suite. It is a suite whose passing means something this month, not only the month you wrote it.