What are LLM evals, and why does production AI need them?

LLM evals are repeatable tests that score a language model system’s outputs against cases and criteria your team agrees on. In production, they decide whether a new prompt, model or version may replace the one in service, or has to wait.

An LLM eval is a fixed set of test cases with agreed criteria, run on every change, that tells you whether the system got better or worse.

Without evals, every error in a pilot reopens the same debate: is it good enough? With them, the question has a number and a date. This is why the evaluation set is one of the six sections of a production definition, and why the absence of one is among the reasons AI pilots stall.

Key takeaways

• An eval is a test you can run again. Same cases, same criteria, every change.
• The process owners write the criteria. Engineers run the eval; the people accountable for the result decide what “good” means.
• Use more than one method. Exact answers, rubrics with human review, model grading and task success each catch different failures.
• Evals do not stop at go-live. They gate every release and score a sample of live work.

What is an LLM eval?

An eval has three parts: a set of cases (real inputs, with the expected answer or the rule the answer must follow), a way to score each output, and a threshold agreed in advance. Run it on version A and version B of the same system, and you can say which one is better on the work that matters to you.

A public benchmark answers a different question. It compares models on general tasks. Your eval compares versions of your system on your cases.

What do LLM evals measure?

Four things, in the order a business owner usually asks about them:

What	The question it answers	Example of a criterion
Quality	Is the output right for this case?	Matches the reviewed answer, or follows the written rule
Safety and policy	Does it stay within what it may say or do?	No answer outside the approved scope
Latency	Is it fast enough for the people using it?	A draft ready before the next step in the process
Cost	Can the operation afford it at volume?	Cost per completed task within the budget line

NIST’s AI Risk Management Framework makes measurement one of its four functions, next to Govern, Map and Manage. Its Generative AI Profile (NIST AI 600-1) names confabulation, confidently stated but false content, among the risks specific to generative systems. An eval set is where that risk becomes a test case.

Which evaluation methods exist?

No single method catches everything. A production eval set usually combines several:

Method	What it measures	When to use it	Its limit
Reference-based	Distance from a known correct answer	Extraction, classification, fixed answers	Needs a reviewed answer for every case
Rubric with human review	Whether an expert accepts the output	Judgment calls, drafts, decisions	Slow and costly, so it runs on samples
Model-graded	A second model scores the output against a rubric	Open-ended answers at volume	Must be calibrated against human review
Task success	Whether the end-to-end task completed	Agents and multi-step workflows	Says that it failed, not always why

Model grading deserves a caution. The authors of Judging LLM-as-a-Judge found that strong models can approximate human preferences, and also documented position, verbosity and self-enhancement biases. Calibrate the grader on a human-reviewed sample before trusting its scores. Broad benchmark projects such as HELM show the same principle at scale: a model is judged on several measures, not on accuracy alone.

How do you build an eval set with the process owners?

This is where most of the value sits, and where engineers alone cannot finish the job. In our engagements, the eval set is built during the diagnosis with the accountable business and technical owners:

1. Collect real cases from the work the system will do, including the hard and rare ones.
2. Have the owners write what a correct output is for each case, so “correct” means what the business means.
3. Agree the thresholds for each measure before any result is seen.
4. Freeze the set at signature, so later results are measured against the same cases.

The first two weeks of a forward-deployed engagement end with that set frozen inside a signed production definition. What the engineer owns in those weeks is described in what a forward-deployed engineer does in the first 30 days.

How do evals run in production?

Evals become a release gate. Every change to a prompt, model, retrieval source or version runs against the frozen set; it ships only if the agreed thresholds hold, and the failing cases are listed when it does not. Once the system is live, a sample of real work is scored the same way, so drift shows up as a number before it shows up as a complaint. How evals fit with the other gates is set out in from AI pilot to production.

What are common mistakes?

• Testing on the examples used to build the prompt. The system passes its own homework.
• Criteria written after the results. Every disappointing score then becomes a debate about the criteria.
• One aggregate score. An average hides the case type that fails every time.
• Evals that stop at go-live. The model, the data and the process change after launch; the eval set has to keep running.

What should you do next?

If your pilot has no frozen eval set yet, start there: real cases, owners’ answers, thresholds agreed in advance. If you want help building it and taking the system to production, tell us about your pilot.