To evaluate AI agents, score whole tasks, not single answers: did the agent reach the right end state, with the right tool calls, inside the rules and limits you set, and did it hand over to a person when it should have. Before go-live you run a fixed set of tasks in a sandbox; after go-live you sample live runs and score them against the same criteria.
Agent evaluation is the practice of running an AI agent through complete tasks and judging the outcome, the actions it took and when it stopped, against criteria agreed before the test.
What an eval set is, and how methods such as rubric review or model grading work, is covered in what are LLM evals. This article is about what changes once the system acts.
Key takeaways
- Judge the end state. What changed in your systems matters more than how the agent explained itself.
- Check the path too. A right outcome reached through a wrong tool call is a risk waiting to happen.
- Run each task more than once. An agent that succeeds once and fails the next time is not ready.
- Measure escalation. Knowing when to hand over is part of the job.
What makes agent evaluation different?
An answer can be read and graded. An agent takes several steps, calls tools, changes records and may talk to people. Three things follow:
- The outcome lives in your systems. The benchmark τ-bench scores an agent by comparing the database state at the end of a conversation with the goal state, rather than reading the transcript.
- The same task can go differently twice. τ-bench also proposes a reliability metric over repeated trials, because agents that pass a task once can fail it on the next attempt.
- The environment matters. AgentBench tests models across distinct interactive environments and reports long-term reasoning, decision-making and instruction following as the main obstacles to usable agents.
What should you measure?
| Metric | What it tells you | How to measure it |
|---|---|---|
| Task success | Did the work get done correctly? | Compare the end state with the expected state |
| Tool-call correctness | Were the right tools called, with the right arguments, in an acceptable order? | Check each call against the task’s allowed path |
| Policy compliance | Did it stay within your rules and permissions? | Rule checks on every action and message |
| Cost and time per task | Is the cost per task sustainable at your volume? | Tokens, tool calls and minutes per completed task |
| Escalation | Did it hand over when it should, and only then? | Share of tasks passed to a person, and whether each handover was justified |
How do you build the test set?
Build it from real work, with the people who own the process:
- Typical tasks, written as a request plus the end state the owners expect.
- Hard and rare tasks, including the ones where the right answer is to stop and ask a person.
- Adversarial tasks, where a document or message the agent reads contains instructions it must not follow.
- The allowed path for each task: which tools may be called, and which actions need a person first.
Sensitive actions should already sit behind approval gates for sensitive automated actions; the test set checks that the agent stops at them.
How do you test before go-live?
Run the full set in a sandbox that mirrors the systems the agent will touch, with test data and the same permissions it will have in production. Repeat each task several times and look at the spread, not only the best run. Then run the adversarial tasks: the OWASP Top 10 for LLM Applications lists prompt injection first, and an agent that reads outside content is exposed to it by design.
The thresholds for each metric belong in the production definition, agreed before the first run, so a result is a pass or a fail and not a debate.
How do you evaluate after go-live?
Live work keeps changing: new kinds of requests, new data, a model update upstream. After go-live:
- Score a steady sample of live runs with the test set’s criteria.
- Review every escalation and every refused approval; they show where the agent’s limits are.
- Re-run the full set before any change to the model, prompt, tools or permissions.
When should an agent lose autonomy?
Write the conditions down before go-live, with the owners: for example, task success falling below its threshold on live samples, a policy breach, or an unexplained jump in cost per task. When one is met, the affected actions go back behind a gate until the cause is fixed and the test set passes again. Autonomy is earned on evidence, and it can be withdrawn on evidence.
What should you do next?
If you have an agent that works on test examples and now has to act on live systems, start with ten real tasks and their expected end states. If you want help building the evaluation and taking the agent to production, see our forward-deployed engineering work or tell us about your project.
Questions
- How do you test AI agents?
- Give the agent complete tasks in a sandbox that mirrors your systems, check the end state each task leaves behind, and repeat each task several times to see whether the result holds.
- What metrics measure AI agent performance?
- Task success, correct tool use, policy compliance, cost and time per task, and how often the agent hands a case to a person. One score cannot carry all of them.
- How do you test an agent against prompt injection?
- Plant instructions inside the documents, emails or web pages the agent reads, and check that it neither follows them nor takes an action it was not asked to take.
- How often should you re-evaluate an agent?
- Before any change to its model, prompt, tools or permissions, and on a steady sample of real runs while it is in service.
Bring us the problem.
Book a call