Evaluation and quality gates

Without evaluation, every change to an AI system is a guess with a demo attached. The team argues about whether the new prompt is better, someone tries four examples, and the version that ships is the one whose author argued hardest. An evaluation harness turns that into a number that either went up or down. It is the single practice that separates a system you can maintain from one you can only admire.

What we actually do

• Building the labelled set with your domain experts, from your real examples, before implementation starts
• Splitting the set by category so accuracy in a large easy category cannot mask failure in a small expensive one
• Holding a set back, untouched, so the reported number is not the number the system was tuned against
• Thresholds agreed in writing at the start of the engagement, with the consequence of a miss agreed at the same time
• Gates in CI: a prompt change, a retrieval change or a provider change is a pull request with a score attached
• Model graded evaluation where the criterion is genuinely subjective, calibrated against human labels rather than trusted on faith
• Scheduled runs against production traffic samples, so decay shows up as a ticket rather than as a support escalation
• Regression cases added from every production failure, so the same fault cannot ship twice

What we instrument

Measurements, not results. These are the readouts we put in place so that you end up with numbers about your own system. There are no values on this page because a value here would be somebody else’s.

• Pass rate per category, and the movement since the last run
• Agreement between model graded scores and human labels on a sampled subset
• Failure taxonomy counts: wrong retrieval, wrong reasoning, wrong format, refused
• Score distribution on the held back set, not just the mean
• Number of production incidents that had a matching regression case before they happened

Where the gate sits

Where this stops working

• An evaluation set is only as good as the labels in it, and labelling is your domain experts' time. We can build the harness in days and the set takes weeks of somebody who knows the answers
• Some outputs have no single correct answer. There the harness measures constraint satisfaction and preference against a rubric, and we will not dress that up as accuracy
• A harness catches the failures you thought of. It is a floor under quality, not a ceiling on risk, which is why guardrails and monitoring exist alongside it

What we work with

Named as plain text. None of these is a partnership, an endorsement or a default: the right one is chosen per engagement, usually the one your team already runs.

• Harnesses built in your own test framework, run in your CI
• Tracing and evaluation tooling including LangSmith and OpenTelemetry
• Labelling workflows in tools your domain experts already use
• Statistical checks on score movement, so noise is not read as improvement