Public benchmarks only shortlist a model; your own cases decide whether to ship it. Here is the dataset, the scorers and the CI gates that catch real drift.
The change that broke something you never saw
You bump the model to the next point release. The contract is unchanged, latency improves, the four prompts you always paste into a scratch script look fine. Three weeks later a ticket arrives: the assistant is quoting a refund window for a plan tier that has none documented. It used to say it did not know. Now it guesses, in the same house tone as every correct answer it has given.
Nothing caught it because nothing was watching. Every knob on an LLM system is a global change with no blast radius you can reason about statically. A new model version, a reworded prompt, a chunking change in the retriever, a temperature tweak, a different quantization of the same weights — any of them can improve the average case while destroying a category of behaviour that worked. No type error, no stack trace. The output is still fluent; it is just wrong now.
Manual spot-checking does not close the gap, and it degrades as the system improves: you test the cases you remember, which are the ones that broke recently, which are the ones you already fixed. Human memory is a terrible index into a regression suite.
If you self-host, this lands harder. On a hosted API someone else usually notices first. When you own the weights you own the version, the quantization, the runtime and the sampling parameters, and you will change all of them — for a new GPU, for cost, for throughput. Every one is a regression only you can detect.
The fix is boring: treat evaluation as regression testing. Cases with expected behaviour, run automatically, failing the build when the system gets worse. Here is how to build one.
Public benchmarks will not do this for you
A public benchmark score tells you almost nothing about whether a change is safe to ship.
They measure general capability on tasks that are not yours: your workload is a narrow slice, and aggregate scores average over a distribution that does not resemble it. They are contaminated, since test items leak into training corpora and you cannot audit what a model has seen. And aggregate improvement is compatible with specific degradation — a model better on average can be worse on your workload, and averages are good at hiding that.
Benchmarks are a shortlisting tool: use them to narrow ten candidates to three, and your own suite to decide which one you ship.
Build the dataset first
Everyone skips this, because picking metrics feels like engineering and collecting cases feels like data entry. The metric sets how precisely you measure; the dataset sets whether you measure the right thing at all. A mediocre metric over an honest dataset beats a sophisticated one over invented cases.
Harvest, do not invent
Cases written by a developer imagining a user are cleaner, better spelled and more on-topic than anything a real user types. They test a system that does not exist. Log inputs, outputs, retrieved context and configuration for every production request, subject to your retention and privacy rules, then build a one-command path from a logged request to a test case. Promoting a real failure has to take a minute, or it will not happen.
Make every bug a permanent case: a report of bad behaviour is not closed until the failing input sits in the suite with its expected behaviour recorded. A year of that habit encodes everything your system has got wrong, which describes your risk surface better than anything you would design up front.
The cases everyone forgets
Harvested traffic is biased toward the happy path, so add the rest deliberately:
- Edge cases. The longest input anyone has sent and the shortest, mixed languages, odd unicode, content at the edge of the context window.
- Adversarial inputs. Injection attempts, instructions embedded in retrieved documents, requests to reveal the system prompt.
- Empty and malformed inputs. Whitespace only, truncated JSON, a required field the caller omitted — these fail spectacularly because nobody tests them.
- Negative cases. Where the correct behaviour is to refuse, decline, ask a clarifying question, or say it does not know.
That last category deserves emphasis because almost every suite under-weights it. If every case has a correct answer, your suite rewards answering, you tune prompts toward answering, and you ship a system that answers everything — including what it has no grounds to answer. The refund-window bug above is a negative-case failure, invisible to a suite made of questions with answers.
How many, and which ones
Fewer than you think, chosen more carefully than you want to. A few dozen cases that each probe a distinct behaviour catch more real regressions than thousands of generated variations of three prompts. Volume looks rigorous in a report and buys redundancy, long runs and a bigger bill.
Stratify instead. Enumerate the behaviours you care about — factual lookup, reasoning over policy, refusal, escalation, tone under hostility, structured output, missing context — and cover each, so a failing run says "refusal dropped, everything else held." That is a diagnosis. If eighty percent of your cases are factual lookup, your headline number is a factual-lookup number in disguise and a serious refusal regression moves it a point at most.
Hold a set back
Your prompts will drift toward your eval. You look at failures, adjust the prompt, re-run; do that thirty times and you have fitted the prompt to those cases. The suite passes, the system is not better. So split the dataset: a development set you iterate against freely, and a holdout you run rarely and never inspect case-by-case while tuning. If development scores climb while the holdout sits still, you have been overfitting. Rotate fresh cases in so it keeps tracking real traffic.
Deciding what to measure
Organise this by how you score. The mechanism sets cost, reliability, and how much a small movement can be trusted.
Deterministic checks
Exact match. Schema validity. Regex and structural assertions. Does the generated code compile, do the generated tests pass. Required and forbidden substrings. Numeric tolerance on an extracted value.
Cheap, fast, perfectly reliable, and they mean the same thing in six months as today. So: use a deterministic check wherever the task allows one. If your system emits structured output, schema validity is a hard gate on every case, not a metric with a threshold. Teams reach for judge models because the task "is open-ended" — usually half of it is not. Assert on that half and hand only the remainder to something that can be wrong.
Reference-based similarity
Comparing output to a gold answer by string overlap or embedding similarity works where a canonical answer exists and is a trap otherwise. The weakness is structural: a correct answer phrased differently scores poorly. "Fourteen days from delivery" and "two weeks after receipt" are the same answer and will not look like it to a string metric. Embedding similarity softens that and adds its own problem, since two texts on a topic score high whether or not they agree. Use it for translation or normalised extraction, never as a build gate on its own.
LLM-as-judge
For open-ended quality — is this helpful, is the tone right, is the refusal appropriate — a model grading model output is the only thing that scales. It is also an instrument with well-documented biases, and treating its score as ground truth means deciding confidently from a broken ruler.
The failure modes: position bias, where judges favour one slot regardless of content; verbosity bias, where longer, hedged answers score higher independent of correctness, dangerous because you then tune toward padding; self-preference, where a judge prefers text from its own family; poor calibration, where a one-to-ten scale returns sevens and eights for nearly everything; and nondeterminism, since the judge is a sampled model too.
What helps:
Write a rubric with concrete criteria. "Rate quality one to five" produces noise. Name the dimensions — is every claim supported by the context, is the disclaimer present, does it speculate, is the format correct — and grade each separately, describing what each grade looks like. Give few-shot examples of each grade — a real five and a real two, with reasoning. That anchors the scale better than adjectives.
Prefer pairwise comparison to absolute scoring. "Which of these two better satisfies the rubric" is answered far more consistently, and it fits regression testing, where you always compare against a baseline. Randomise option order on every such call; where affordable run both orders and treat disagreement as a tie, which doubles as a health metric for the judge.
Use a different model as judge than the one under test. Non-negotiable if the comparison is to mean anything.
Validate the judge against human labels before trusting it. Sample cases, have a human grade them against the same rubric, measure agreement. If the judge disagrees with your humans on a quarter of cases, its scores cannot support a gating decision. Re-run whenever the rubric or judge changes.
Behind all of it: the judge is a component of your system and must be evaluated like one. It has a version, a rubric, a measurable error rate, and it regresses when you upgrade it.
Human review
Human judgement is ground truth and expensive, so spend it on calibration rather than routine runs: labelling the sample that validates your judge, adjudicating scorer disagreements, checking the suite still reflects real usage.
Retrieval systems: score the stages separately
An end-to-end score says the answer got worse and nothing about why. Did the retriever stop returning the right document, or did the generator stop using it?
Score retrieval on its own terms, with labelled relevant documents per query: recall at k (are the documents needed to answer in the top k), precision at k (how much junk crowds the context), mean reciprocal rank (how far down the first relevant document sits — models weight early context differently from late).
Score generation given retrieved context: groundedness (every claim traceable to the context, hallucinations counted), answer relevance, and correct behaviour when the context lacks the answer — a negative case, and where most RAG systems fail unnoticed. Run it against fixed context as well as live retrieval, or a reindex and a prompt edit look identical in your results.
The harness itself
Five components, none of them clever.
A case store, version-controlled and reviewed in pull requests like any other code — one file per case, a directory per category. Not rows in a database edited through a web UI, because you need diffs, blame and bisect.
id: refund-window-undocumented-tier
category: refusal
input: "What is the refund window for the Platinum plan?"
context: [fixtures/kb/refunds-standard.md]
expect:
must_not_match: "[0-9]+ (days|weeks|months)"
must_contain_any: ["do not have", "not documented", "cannot confirm"]
judge_rubric: rubrics/refusal-quality.md
judge_min_grade: 4
A runner that executes cases against a pinned configuration, with concurrency, timeouts, retries on transport errors but never on bad content, and subsetting by tag or id. One case locally must be as easy to run as everything in CI.
Scorers as small, independently testable functions returning a number and a reason — deterministic ones are pure functions with their own unit tests, judge scorers the same shape with a model call and a pinned rubric inside.
A results store, append-only, one record per case per run. Homegrown harnesses fall over here: without history you cannot separate a regression from noise, and baseline comparison is the whole job. Then reporting that answers three questions fast: what changed since baseline, which categories moved, which cases flipped.
The rule that makes it work: every run records its complete configuration, or results are not comparable. Model identifier, revision or weight digest, quantization, serving runtime, prompt template hash, index version, top-k, reranker, temperature, seed, max tokens, and the judge's own model and rubric version. Anything unrecorded is a variable you cannot control for.
Nondeterminism is not a bug you can fix
Temperature zero reduces variance; it does not eliminate it. Floating-point reduction order varies with batch composition, kernels differ across hardware and driver versions, expert routing can depend on what else is batched, and hosted endpoints move you between machines. A seed helps only where it is honoured end to end.
Design for it. Run each case several times — three to five repeats shows the shape — and report a distribution, not a point. Use pass-rate thresholds rather than single-run pass/fail: "passes four times in five" is a stable statement about a stochastic system, where "it passed" is a coin flip treated as fact. Establish your noise floor by running one configuration several times and measuring the spread; a tighter threshold produces flaky failures that teach the team to ignore the suite. And escalate on persistence: one slight dip is noise, the same category down across three consecutive runs is a regression even if no run crossed the line.
Wiring it into CI
A suite nobody runs automatically is a document, not a test.
Split by cost. Per pull request, run a fast subset: every deterministic check, since they cost nothing, plus a stratified sample of judge-scored cases sized to finish in a few minutes. Nightly or pre-release, run everything with full repeats, plus the holdout. Any prompt, model, retrieval or judge change triggers the full suite regardless of schedule.
Gate on relative regression, not absolute score. Absolute thresholds are guesses that go stale. The question is whether this change made things worse than the current baseline: compare against the last main-branch run at the same configuration and fail on a drop beyond your noise floor. Gate per category too, since that catches what a headline average absorbs.
Set thresholds so the suite survives. A gate that fires on noise gets disabled within two weeks, first with a skip flag, then permanently. Start loose: hard-fail on deterministic failures and unambiguous category drops, warn on everything else with a trend line, and tighten as you learn the real variance. A suite that catches eighty percent of regressions and is trusted beats one that catches ninety-five percent and is routinely overridden.
Treat an eval failure as a build failure, same severity as a failing unit test. If the answer to a red eval is habitually "that one is flaky, merge it," you have built something worse than nothing: a green check certifying nothing, and a team that has stopped reading it. Fix the flakiness, fix the threshold, or delete the case. Allow overrides, but make each cost a named approver and a written reason in the run history. Overrides that need a sentence get used rarely; overrides that need a click get used always.
Controlling the cost
Cache by configuration hash. If model, prompt, retrieval settings, sampling parameters and case content are unchanged, the result is reusable — key on config hash plus case hash and most pull requests pay for almost nothing. Sample for speed: a stratified sample gives signal in minutes; the full suite runs nightly, where an hour is free. Use small cheap models for smoke tests, not to measure quality but to check that the harness runs, prompts render and schemas parse before the expensive run. And keep the judge proportional: every case moved to an assertion is a permanent cost cut.
Living with the suite
A suite decays if unattended, because your product and your users change while the cases stay put. Every production bug becomes a case. Retire cases that no longer discriminate: one that has passed every run for a year at full marks is not testing anything, so keep a canary and archive the rest. Review for drift quarterly against real traffic — if half the suite tests a feature two percent of users touch, your gate is protecting the wrong thing. And version the suite alongside the system, so a deliberate behaviour change and its new expectation land in one pull request.
A build order
Build it in the order that earns trust.
An afternoon. Twenty to thirty real cases in a YAML file, harvested from logs and your bug tracker, and a script that runs them against your current configuration and prints pass and fail. Deterministic checks only, plus at least five negative cases where the right answer is a refusal. Commit it. That is a regression suite, and it catches more than you expect, because most regressions are crude.
The first week. Wire it into CI on pull requests. Record results with full configuration into an append-only store, add baseline comparison so the output says what changed rather than what passed, then repeats and pass-rate thresholds.
The first month. Add a judge with a real rubric for the open-ended categories, validated against human labels before you trust a score from it. Split the dataset and hold part back. If you run retrieval, separate the two stages' metrics.
Ongoing. Caching, per-category gating, a human review queue, and a pipeline that files production failures as cases automatically.
Each stage is useful alone, which is the point. The version running tonight is not a sketch of the real thing — it is the thing, small. Thirty honest cases that run on every pull request and fail loudly catch regressions a beautiful unbuilt harness never will. Start there, and let every bug you ship make it bigger.