Demystifying Evals for AI Agents
Agents fail in production because teams can't measure whether they're actually working.
McKinsey's 2025 data puts 62% of organizations experimenting with AI agents, and 23% already scaling deployments. Gartner predicts more than 40% of agentic AI projects will be canceled by 2027. Not because the agents can't do the work. Because teams couldn't tell whether they were doing it right.
"We ran it a few times and it seemed fine" is not an evaluation strategy when something is making decisions and taking actions at scale. So why does that gap keep showing up?
What makes an agent categorically different from a model you prompt once
A standard model call is almost boring in its simplicity. One input, one output, stateless. You send a message, you get a response, you grade it. The transaction is over.
Agents don't work like that. They're goal-driven, multi-turn, operating in a real or simulated environment, adapting based on what they find. That structure creates four evaluation headaches that flat-out don't exist in the single-prompt world.
Non-determinism. Run the same task twice and you can get completely different action sequences. One test run isn't evidence of anything. It's a single point on a noisy distribution.
Multi-turn error propagation. An agent can pass the early steps cleanly and then quietly unravel. If you only check whether the final answer looks right, you'll miss the compounding failures that produced it.
Tool and memory misuse. An agent can call the wrong tool, ignore context it captured three steps ago, or act on stale information. None of that necessarily surfaces in the final response. The answer can look perfectly reasonable while the underlying reasoning was a mess.
The outcome vs. transcript problem. This is the one that actually bites people. An agent can produce a pleasant, confident final message while having failed in the environment. A flight-booking agent outputs: "Your flight has been booked." Sounds great. But what if no reservation was ever created in the database? Evaluate the text and you declare success. Evaluate the environment and you catch the failure.
The agent's words are not the product. What the world looks like after it acted is the product.
The vocabulary that makes agent evals precise
You can't build a good eval without a shared language. Anthropic's January 2026 engineering post on agent evals has become something of an industry reference point, and the terminology it lays out is worth internalizing. Precision here isn't pedantic. It's functional.
- Eval: a test. Give the agent an input, apply grading logic, measure success. Specifically, automated evals that run during development without real users in the loop.
- Task (or test case): a single test with defined inputs and success criteria.
- Trial: one attempt at a task. Because outputs vary, you need multiple trials per task.
- Grader: the logic that scores performance. Code, an LLM, or a human.
- Transcript (trace / trajectory): the complete record of a trial, including outputs, tool calls, reasoning steps, and intermediate results.
- Outcome: the final state of the environment. Distinct from what the agent said.
- Evaluation harness: the infrastructure that runs tasks, records steps, grades outputs, and aggregates results.
- Agent harness (scaffold): the system that enables the model to act as an agent at all.
That last one deserves a moment. When you evaluate an agent, you are not evaluating the base model in isolation. You're evaluating the model plus its scaffold, together, as a system. Change the scaffold and you've changed what you're measuring. This is why teams sometimes swap in a newer model, run their benchmark, and see no improvement. They might be looking at the wrong variable entirely.
Teams that conflate "transcript" with "outcome," or "trial" with "task," end up designing evals that answer the wrong question. They often don't realize it until something breaks in production.
What to actually measure: the three grader types and when to use each
No single grader type handles all of agent evaluation well. The practical move is to match the grader to the specific claim you're trying to make.
Deterministic / code graders. Start here whenever possible. Did the database record change? Does the reservation exist? Did the right API get called with the right parameters? High signal, no ambiguity, fast to run. If the agent's action produces a verifiable side effect, a code grader should be your first instinct. The limitation is that not everything an agent does is cleanly checkable, and that's when you escalate.
LLM-as-judge. For outputs that require interpretation, a strong model can be prompted to rate quality, correctness, or policy compliance, essentially simulating a human evaluator at scale. Studies of MT-Bench and Chatbot Arena show strong LLM judges achieve over 80% agreement with human preferences, roughly on par with agreement between different human evaluators. Useful for response tone, policy adherence in natural language, and reasoning quality in a transcript.
Human graders. Highest signal, lowest throughput. Irreplaceable in early development for establishing what "good" actually means before you have enough signal to automate. Anthropic's Claude Code eval process started with internal human evaluation trials, which captured basic capability trends and enabled fast iteration before any investment in a comprehensive suite. Human graders aren't meant to run at scale. They're meant to ground-truth everything else.
One concrete starting point from Anthropic: begin with 20 to 50 simple tasks drawn from real failures. Early changes tend to have large effect sizes, so small sample sizes can be sufficient when your agent is still rough.
Why LLM-as-judge fails in specific, predictable ways
LLM judges are not neutral arbiters. They're fallible measurement instruments with documented, systematic biases. Treating them as ground truth is the mistake. Understanding their failure modes is what lets you use them well.
Position bias. Judges disproportionately favor answers that appear earlier in a response set, regardless of actual quality. The order you present options in shapes the score.
Self-preference bias. Research presented at NeurIPS 2024 showed a linear correlation between a model's ability to recognize its own outputs and its tendency to prefer them. A model judging its own generations isn't a neutral evaluator. It's grading its own homework.
Error cascade in agentic settings. In multi-step evaluations, early misjudgments compound. Later decisions rest on flawed earlier ones. A bad call in step two can significantly distort the final score.
Adversarial gaming. Recent research demonstrated that even nonsense responses can receive high scores from a capable LLM judge if they're written in a sufficiently persuasive style. The judge is, at some level, susceptible to being charmed.
Galileo's research found that teams using LLM judges achieve meaningfully better reliability and more comprehensive incident detection compared to those using a single, uncalibrated judge. The problem isn't LLM judges. It's uncritical use of a single LLM judge with no mitigation.
Randomize response positions. Use multiple models as judges rather than one. Calibrate LLM judge outputs against human labels. Combine grader types. The biases are manageable. They just have to be actively managed rather than assumed away.
The pass@k vs. pass^k distinction and what it reveals about production readiness
Most teams skip this metric conversation. It's also the one that tends to bite them hardest.
Two metrics answer two completely different questions.
Pass@k: the probability that at least one of k independent attempts succeeds. It measures peak capability. If you give the agent five shots at a task, does it get it right at least once? This is the standard for code-generation benchmarks.
Pass^k (pass-power-k): the probability that the agent succeeds across all k repeated executions of the same task. It measures consistent reliability. Does the agent get it right every time?
At a moderately high per-trial success rate, these two metrics diverge by more than 60 percentage points. Pass@k looks acceptable. Pass^k exposes fragility. Think about a surgeon who nails the procedure seven times out of ten. Impressive in a research paper. Alarming on your operating table.
At that same moderately high per-trial rate, an agent handling three consecutive tasks of the same type will complete all three successfully only about a third of the time. Fine in a demo. A reliability problem at production volume.
A 2025 survey of 306 AI agent practitioners found reliability issues are the single biggest barrier to enterprise adoption. Practitioners are already responding by restricting agents to shorter, fewer-step workflows just to manage this gap. That's not a design choice. It's a workaround for knowing the agents struggle to sustain pass^k at any meaningful scale.
What public benchmarks measure well, and where they stop being useful
Three benchmarks are worth knowing because each tests something distinct.
SWE-bench Verified. Five hundred real GitHub issues, hand-checked for solvability. Scores on whether the agent's code changes pass the repository's existing tests. Pure technical correctness, no policy dimension. Frontier models moved from 40% to over 80% on SWE-bench Verified in roughly one year, with Claude Opus 4.7 reaching 87.6% as of April 2026. Progress that rapid is impressive and also a signal that saturation is approaching. Evals that differentiated models six months ago may no longer discriminate between good and great.
τ-bench / τ²-bench. Scores whether the agent satisfies a user request while following defined business policies: refund rules, booking change procedures, authorization limits. It's the closest public proxy for customer-facing enterprise agents. GPT-4o achieved roughly 60% on retail tasks and 42% on airline tasks as of 2025. Policy complexity, not just task complexity, drives difficulty. The leading score as of early 2026 is 85% on pass^1, but airline sub-scores remain substantially lower than retail. Domain matters more than headline scores suggest.
WebArena. 812 tasks across five sandboxed websites requiring multi-step navigation: clicking, form filling, cross-page transitions. Tests grounded action in rendered environments rather than API calls. Scores remain well below where SWE-bench has landed, which reflects how much harder visual and navigational grounding is than structured code tasks.
These benchmarks test a fixed distribution of tasks that may have little overlap with what your agent does in your environment. They're useful for model selection and for tracking progress over time. They are not a substitute for evals built against your own failure cases.
The Anthropic Opus 4.5 and τ²-bench episode illustrates this directly. The agent found a policy loophole, technically failed the eval as written, but produced a better real-world outcome. The grader was measuring the wrong thing. A benchmark can only tell you what it was designed to ask. It cannot tell you whether that's the right question for your agent.
How to build an eval suite that reflects the agent you actually deployed
Everything above is scaffolding for this part. The goal isn't to understand evals abstractly. It's to build one that works.
Start with real failures, not synthetic coverage. Begin with tasks drawn from actual breakdowns in development or early production. Synthetic tasks miss the edge cases that real usage surfaces. The first 20 to 50 tasks don't need to be comprehensive. They need to be honest.
Match the grader to the claim. Environmental state changes go to deterministic graders first. Policy compliance and language quality go to LLM-as-judge with documented bias mitigations. Novel failure modes go to human graders to establish ground truth, then get encoded into automated graders once you understand the pattern.
Measure pass^k, not just pass@k. Especially for any task type the agent will perform repeatedly at volume. The best-case metric will make you feel better. The typical-case metric will make your agents better.
Evaluate the model-plus-scaffold system. Not the model in isolation. Changes to the harness or tool definitions change what you're measuring. Test the thing you actually deployed, not a cleaner version of it.
Treat the eval suite as a living artifact. Benchmark saturation is real. Production distributions shift. An eval that no longer discriminates between good and bad runs has stopped being useful. It just hasn't told you that yet.
An eval suite is the internal evidence base that lets a team make a confident deployment call, explain risk to a skeptical stakeholder, and actually know whether the agent got better or whether the bar just got lower. Catching failures and understanding your agent are related, but they're not the same thing. Treating them as interchangeable is how you end up with an agent that looks great on paper and quietly fails the people using it.
