The failure this prevents
Here is the shape of it, and it is the same in almost every team. Someone builds a prototype. It demos well, because the examples in the demo were chosen by the person who built it. It goes to a pilot group. Complaints arrive as anecdotes: it got this one wrong, it was weird about that one. Someone changes the prompt. The complaints change shape. Nobody can say whether the system is better than it was three weeks ago, because nothing was ever measured.
At this point the team is not engineering. It is negotiating. The next change ships because the person proposing it was more persuasive, or more senior, or tried four examples that happened to work. That is not a tooling problem and no observability product fixes it. It is an ordering problem: the measurement was supposed to exist before the thing being measured.
What a usable set actually contains
A set is not a spreadsheet of questions. It is questions paired with what a competent person would consider a correct response, drawn from your real traffic or your real documents, and organised so that the number it produces means something.
- Real inputs, taken from what users actually send or what your documents actually contain, including the malformed ones. Curated inputs measure a system that will never exist.
- Categories, so that the score can be read per category. A system that is excellent on the seventy percent of cases that are easy and useless on the thirty percent that are expensive reports a good average and is a bad system.
- A held back portion that nobody tunes against, kept honest by being kept closed. The number you report should not be the number you optimised.
- Known failure cases from previous attempts, including the embarrassing ones. Every production failure joins the set afterwards, so the same fault cannot ship twice.
- For subjective outputs, a rubric rather than a target string, and a sample of human labels to calibrate any automated grading against.
The threshold conversation is a commercial conversation
The most useful part of building the set is not the set. It is the conversation where you and the buyer agree what number is good enough, and what happens if it is not reached. That conversation is uncomfortable early and catastrophic late.
It also forces a question most projects avoid: what is the cost of a wrong answer here. If a wrong answer is mildly annoying, a lower threshold and a fast path is right. If a wrong answer reaches a regulator, the threshold is high, the escalation rate is expected to be high, and the design is different from top to bottom. Nobody can design the guardrails until somebody has answered that, and the evaluation set is where the answer gets written down.
Then put it in the pipeline
A set that runs when someone remembers is a set that stops running in month two. It belongs in continuous integration, on the same pipeline as the type check and the test suite, so that a prompt change is a pull request with a score attached and a retrieval change cannot merge while quality is down.
It also belongs on a schedule, running against a sample of production traffic, because the two things that decay quietly are the corpus and the input distribution. Neither of those events is a code change, so neither is caught by a gate that only fires on code changes.
The honest limits
A harness catches the failures you thought of. It is a floor under quality, not a ceiling on risk, which is exactly why guardrails, grounding checks and monitoring exist alongside it rather than instead of it.
And the labelling is real work by people whose time is expensive, because the labels have to come from somebody who knows the right answer. That constraint is the reason most teams skip this step. It is also the reason the teams that do not skip it are still shipping improvements in month nine while the others are still arguing about a prompt.