LLM eval harness: how we build one in a day
Every sprint has the same item on day 14: put an eval harness in CI. It's usually the first thing we build that isn't a feature, and it tends to be the thing that should have existed already.
An LLM eval harness is an automated regression suite for a feature built on a language model. Save about 50 real inputs, assert on properties rather than exact strings, run it on every change to the prompt, model, retrieval or chunking, and fail the build when the pass rate drops against main. It takes roughly a day to build and needs no vendor.
Most teams shipping an LLM feature have no automated way to know whether this morning's prompt change made it worse. It isn't negligence — the normal testing instinct breaks on non-deterministic output, so the suite gets deleted and everyone falls back to eyeballing a few queries before deploy.
Here's how we do it, in case it's useful to build yourself.
| 1. Collect cases | 50 real inputs from production logs, including every past failure. |
|---|---|
| 2. Write assertions | Structural, must-contain, LLM-as-judge with a rubric, and cost and latency budgets. |
| 3. Wire into CI | Run on any change to prompt, model, retrieval or chunking. |
| 4. Gate on delta | Fail the build when pass rate drops against main, not when it is below 100%. |
| 5. Pin the model | Use a dated model version so upgrades are deliberate. |
We start with 50 real cases, not synthetic ones
We pull them out of production logs — the 30 most common user inputs, plus every input that has ever produced a bad output. That second group is the half that matters. It turns every past incident into a permanent regression test.
We stop at 50 deliberately. It's tempting to try to reach 500, and the teams that do tend to ship nothing. Fifty catches most regressions, and one more case gets added every time something breaks.
We assert on properties, never on exact strings
This is where the usual testing instinct breaks. assert output == expected fails the moment the model says the same thing in different words, the suite goes red for no reason, and someone deletes it inside two weeks.
So we assert things that survive non-determinism:
Structural
Valid JSON, required fields, correct types, enums in range. This catches more real breakage than we expected when we started doing it.
Must-contain / must-not-contain
The refund answer has to mention the 30-day window; no response may name a competitor. Cheap, deterministic, and it puts the actual business rules in the test suite.
LLM-as-judge, with a written rubric
For the genuinely fuzzy dimensions. We score 1–5 against a rubric we wrote down, using a different model than the one under test. Judge scores belong in the aggregate — per-case they're too noisy to gate on.
Cost and latency budgets
p95 latency and mean tokens per call. We added this after watching a prompt edit triple a bill with nothing else changing.
We run it on four triggers, not one
Any change to the prompt, the model, the retrieval, or the chunking. Only one of those looks like code, which is why a prompt edit in a config file is the silent regression we hit most often.
We gate on the delta, not the score
We don't require 100% pass. That produces a flaky suite everyone learns to ignore, which is worse than no suite. We record pass rate per run and fail the build if it drops more than a few points against main.
What we're after is the trend line, visible on every PR.
We pin the model version
gpt-4o and gpt-4o-2024-11-20 are not the same thing. On a floating alias, a provider can change your product's behaviour on a random Tuesday with no deploy on your side. We pin the dated version and make the upgrade a PR that runs the harness. It's one line of config and it prevents a whole category of 3am problem.
LLM eval harness FAQ
What is an LLM eval harness?
An LLM eval harness is an automated test suite for a feature built on a language model. It replays a fixed set of saved inputs through the system on every change and checks the outputs against assertions that tolerate wording differences, so you find out that a prompt edit made things worse before your users do.
How many test cases do you need for LLM evals?
Around 50 is enough to start. We use the 30 most common real user inputs plus every input that has previously produced a bad output. Teams that hold out for 500 cases usually ship nothing at all, and a 50-case suite running today catches far more than a perfect suite that never gets built.
Why can't you use normal assertions to test LLM output?
Because the model is non-deterministic. An exact-match assertion fails the moment the model expresses a correct answer in different words, the suite goes red for no real reason, and within about two weeks somebody deletes it. Assert on properties instead: structure, required content, forbidden content, and cost and latency budgets.
What should trigger an LLM eval run in CI?
Four things: a change to the prompt, the model, the retrieval, or the chunking strategy. Only the last of those tends to look like code, which is why a prompt edited in a configuration file is the most common source of silent regression.
Should an eval suite require a 100% pass rate?
No. A suite that demands perfection on non-deterministic output becomes flaky, and a flaky suite is ignored, which is worse than having none. Record the pass rate for each run and fail the build when it falls more than a few points below main. The trend line is the signal.
Why pin the model version?
Because a floating alias such as gpt-4o resolves to whatever version the provider is currently serving, so your product's behaviour can change with no deploy on your side. Pinning a dated version such as gpt-4o-2024-11-20 makes every upgrade an explicit pull request that runs the harness first.