Choosing an Agent Evaluation Framework
Matching evaluation frameworks to agent architectures prevents costly deployment failures.
Agent evaluation frameworks have splintered into a handful of genuinely different tools, and the split is not arbitrary. It maps directly onto how agents are built: how many steps they take, whether they call tools, whether they retrieve context before reasoning, whether they run inside one orchestration framework or many. This piece walks through the dimensions that separate these tools and matches them to the architectures they actually serve, because picking the wrong one does not just waste a sprint. It leaves a team unable to explain why an agent failed, which is a much worse place to be than simply having no automated tests at all.
Something has changed in how "correct" gets defined for these systems. A single-turn model either answers a question right or it doesn't. An agent plans, calls tools, holds state across steps, talks to external services, and produces a trajectory whose effects compound. A synthesis published on arXiv (2604.19818) frames this as a shift from asking "did the model answer correctly?" to asking "did the system behave acceptably throughout execution?" That's a harder question, and most evaluation tooling still isn't built to answer it. A final-answer metric can show green across the board even when step four called the wrong API, step six retried the same failed action three times, and step eight only recovered by accident.
The scale of the gap is visible in benchmark numbers, too. Stanford HAI's 2026 AI Index reported that Terminal-Bench success on real-world tasks jumped from 20% in 2025 to 77.3% in 2026, a genuinely startling year-over-year gain. But the same report's authors flagged that capability is outpacing the field's ability to measure and govern these systems. Agents are getting better faster than anyone can verify how or why. Compounding that, the Agentic Benchmark Checklist (ABC) work found that small, seemingly innocuous choices in benchmark design (how a task is scored, what counts as task completion, how retries are handled) can swing reported agent performance by up to 100% in relative terms. Two benchmarks measuring "the same" capability can produce wildly different verdicts depending on design choices nobody outside the benchmark's authors ever scrutinizes. None of this is really a tooling shortfall. It's a mismatch between how agents behave and how evaluation was designed to measure much simpler systems.
What a production deployment gap actually looks like
McKinsey's 2025 research put a number on the disconnect: 62% of organizations were experimenting with or piloting agents, 23% had scaled agents in at least one business function, but no single function crossed roughly 10% in reported scaling. Piloting is cheap. Scaling is where most efforts stall out, and the stall isn't evenly spread, it shows up everywhere at once.
The pace of piloting tells its own story. Enterprises running agentic AI pilots nearly doubled in a single quarter, from 37% in Q4 2024 to 65% in Q1 2025. Production adoption did not follow at anywhere near that rate. Interest accelerated; deployment didn't catch up.
Only about 10% of enterprises get generative AI into production successfully, and inadequate evaluation is frequently cited as a contributing reason the rest don't. Gartner's forward-looking estimate puts a similarly grim number on what happens next: over 40% of agentic AI projects are predicted to be canceled by 2027, and the inability to evaluate deployed agents systematically is named as a contributing cause. This is not a story about model quality catching up over time. It's a story about measurement infrastructure lagging behind deployment ambition.
Practitioner reports on successful AI product teams put evaluation at 60 to 80% of total development time, and most of that time doesn't go toward writing automated checks. It goes toward understanding failures: reading transcripts, tracing where a chain of tool calls went sideways, figuring out whether the agent was wrong or just unlucky. Picking the wrong evaluation framework doesn't just slow this process down. It removes the ability to do it at all, because the framework that can't see intermediate steps can't tell anyone what actually broke.
The dimensions that actually distinguish agent evaluation frameworks
Four things separate agent evaluation from simple output-checking, and any framework worth adopting needs to reach at least some of them.
Tool call correctness asks whether the agent picked the right tool and invoked it with the right arguments at each step, not just whether the final answer happened to land correctly. Trajectory quality asks something subtler: was the reasoning path coherent, or did the agent stumble into the right answer through a sequence of wrong turns that happened to cancel out? Multi-turn consistency checks whether context carries properly across a conversation and whether the agent can recover after an earlier mistake instead of compounding it. Intermediate failure visibility is the practical payoff of all three: can someone point to the exact step in a ten-step chain where things first went wrong, rather than shrugging at a failed final output with no idea which upstream decision caused it?
Underneath all four sits a genuinely hard problem: agentic systems are non-deterministic. Research on agentic systems documents variability in execution paths, tool selection, and memory retrieval patterns between runs of what is nominally the same agent on the same input. Uncertainty doesn't stay contained to one component. It propagates through tool invocations, through memory operations, through however many agents are coordinating in a multi-agent setup. An evaluation framework that assumes deterministic, repeatable runs is going to misattribute a lot of failures to the wrong cause.
Enterprise cost adds a dimension that's easy to overlook when a team is focused purely on accuracy. An evaluation of six leading agents across 300 enterprise tasks found that optimizing for accuracy alone produced agents that were 4.4 to 10.8 times more expensive to run than cost-aware alternatives delivering comparable performance. Chasing the highest score on a leaderboard can mean shipping something nobody can afford to run at volume.
A four-pillar structure from production CloudOps experience breaks agent failure modes down further: LLM, Memory, Tools, Environment. Each pillar fails in its own particular way, and none of those failure modes show up cleanly in an end-to-end score. A memory retrieval bug and a tool-selection bug can produce the identical wrong final answer, and only pillar-level visibility tells them apart.
That gives a practical sorting logic for everything that follows: how many steps the agent takes, whether it calls tools, whether trace-level failure attribution matters more than an aggregate pass/fail score, whether the agent is conversational or pipeline-driven, and whether the priority is CI/CD gating or live production monitoring.
MLflow for teams that need end-to-end trace evaluation in a unified platform
MLflow is the most widely adopted open-source AI engineering platform by a wide margin, pulling a substantial number of monthly downloads on PyPI. Its evaluation layer is built on a specific architectural bet: scorers see the complete execution trace, not just the final output. Tool calls, reasoning chains, planning decisions, all of it is available to the scoring function. That's what makes it fit for multi-step agents where the interesting failures happen in the middle of the run, not at the end.
Agent GPA (Goal-Plan-Action) scorers, available through third-party integrations, cover common agent evaluation patterns. Teams with more idiosyncratic needs can write custom scorers directly in Python. On the judge-alignment side, MLflow applies research-backed algorithms (GEPA, MemAlign) to tune LLM judge prompts against human labels, closing the gap between what an automated judge says and what a human reviewer would actually flag. Without that step, automated judge scoring tends to drift from human intuition in ways that are hard to catch until someone spot-checks a batch of transcripts.
MLflow's evaluation layer is designed to handle complex agent workflows, with test coverage extensible beyond whatever's sitting in a labeled dataset. That tight integration between tracing and evaluation means a regression caught in the wild can inform testing rather than remaining a one-off incident report.
The tradeoff is setup complexity. MLflow's breadth (tracing, evaluation, prompt optimization, governance, all under one platform) is exactly what makes it overkill for a team that just needs to confirm a single-turn output looks reasonable.
DeepEval for teams running evaluation inside their existing CI/CD pipeline
DeepEval reads like pytest because it's meant to. Metrics run as test cases inside the same pipeline that already runs unit tests, which makes it a natural fit for teams where evaluation means "does this still work the way it worked last week."
That framing matters most for agents whose primary risk is silent regression rather than novel failure modes. DeepEval supports multi-turn evaluation, but visualization requires Confident AI, a freemium platform with a free tier and paid plans for higher usage, which is worth factoring in for teams committed to staying open-source end to end. It also doesn't ship built-in conversation simulation or LLM judge alignment, so calibrating an automated judge against ground truth is left to the team.
MLflow's native integration with DeepEval closes part of that gap: DeepEval metrics can run as MLflow scorers, with results tracked in the MLflow UI. That lets a team keep DeepEval's CI/CD-native testing style while getting production monitoring without adopting a second, unrelated toolchain. For engineering teams that think about evaluation the way they think about unit tests, blocking a deploy on a failed check, DeepEval fits naturally. It's less suited to teams already wrestling with production drift or the kind of intermediate-trace debugging MLflow is built for.
Ragas for agents whose primary retrieval step determines answer quality
Ragas is scoped narrowly and deliberately: retrieval-augmented generation metrics, specifically context relevance, faithfulness, and answer relevance. That narrowness is the point. When a RAG agent's retrieval step pulls the wrong context, no amount of downstream reasoning quality saves the output, and a generic accuracy metric will often miss this entirely because the model reasoned perfectly well over garbage.
The stakes here aren't abstract. A mid-sized e-commerce firm building an agentic supply chain optimizer watched infrastructure costs climb from $5,000 a month in prototyping to $50,000 a month in staging, a tenfold jump, driven by unoptimized RAG queries pulling in far more context than the task actually needed. Retrieval-specific metrics of the kind Ragas provides would have surfaced that pattern before it hit staging, not after the bill arrived.
Ragas is not built for conversational depth. Multi-turn evaluation is limited, and there's no conversation simulation, so agents that carry context across many turns aren't its target use case. It does integrate natively with MLflow as a third-party scorer, so it can sit inside a broader trace-evaluation setup rather than standing alone. The fit is narrow but sharp: agents built on a retrieval backbone, where "did we retrieve the right thing" is the question that actually determines whether the agent succeeds.
Arize Phoenix for teams extending existing ML observability to LLM evaluation
Phoenix comes from ML observability rather than from agent tooling specifically, and that lineage shows. It operates under the ELv2 license, with production monitoring and dataset management layered in alongside evaluation.
The natural fit is a team that already runs ML ops workflows, tracking model drift, watching data quality, and is now adding agents into that same observability plane rather than standing up a separate system just for LLMs. Phoenix is designed for observability workflows, with multi-turn evaluation support that varies depending on the use case. Teams leaning on automated judges will need to handle calibration work themselves. Like Ragas and DeepEval, it integrates natively with MLflow.
The best fit is an ML platform team inside a larger enterprise where agent evaluation needs to live next to existing model monitoring, not in a separate silo that the platform team has to maintain twice.
LangSmith for teams already running LangChain or LangGraph in production
LangSmith is the observability and evaluation layer built into the LangChain ecosystem. It isn't independently open-source: there's no OSI-approved license covering the platform itself, though the client SDK is MIT-licensed. The backend and UI stay proprietary. What it does have is depth of integration with LangChain and LangGraph that nothing else on this list can match, because it was built for that stack specifically.
Its download numbers need a caveat. LangSmith shows a monthly download count on PyPI that dwarfs that figure, which sounds enormous next to MLflow's. But LangSmith installs automatically as a dependency of the langchain package itself, a footnote flagged explicitly in MLflow's own source material. That inflates the count considerably; it isn't a clean adoption signal in the way MLflow's download figure is.
LangGraph itself is recommended specifically for stateful multi-agent orchestration and long-running workflows. Deep Agents, a separate harness alongside LangGraph, is recommended for autonomous long-running tasks specifically. LangSmith sits above both, adding enterprise-grade observability and evaluation once the orchestration layer is in place. It supports dataset management, multi-turn evaluation, human feedback collection, and online monitoring, with visualization built in.
The catch is coupling. LangSmith's evaluation machinery is tied closely to LangChain's tracing format, so teams running CrewAI, OpenAI's Agents SDK, or Google's ADK will find it far less native, if usable at all in the way LangChain users experience it. The clean fit is a team already committed to LangChain or LangGraph that wants evaluation, tracing, and orchestration sharing one data model without extra integration work bolted on afterward.
Promptfoo's acquisition status and what it means for vendor-neutral red-teaming
Promptfoo built its reputation as an open-source CLI and library for evaluating and red-teaming LLM applications, with a specific value proposition: vendor neutrality across model providers. That positioning is worth naming directly, because it's the thing now in question.
As of the March 2026 announcement, Promptfoo agreed to be acquired by OpenAI. OpenAI has confirmed the project stays open source under its current license, so nothing changes at the code level immediately. But the project is no longer independent, and independence was a meaningful part of why teams chose it in the first place.
For teams evaluating non-OpenAI models, this is worth weighing directly rather than glossing over. The tool's governance now sits with one model provider, and that's a different risk profile than an unaffiliated open-source project, regardless of how the license reads on paper. This doesn't disqualify Promptfoo. It changes the calculus for teams building on non-OpenAI infrastructure, or for teams with procurement rules that specifically require vendor-neutral tooling. Teams using it mainly for red-teaming and adversarial prompt testing are probably less exposed to this shift than teams that had built Promptfoo into the primary evaluation harness across a genuinely mixed-model stack.
How the agent's architecture should drive framework selection
The right question was never "which framework is best." It's "what does this agent actually do, and which failure modes matter most here?" Framed that way, the decision becomes a series of concrete axes rather than a popularity contest.
Step count is the first axis. Single-turn output checking is a fundamentally different problem than multi-step trajectory evaluation, and the latter requires trace-aware scorers, which narrows the realistic field to MLflow and LangSmith. Tool use is the second: any agent invoking external tools needs tool-call correctness metrics, because a framework that only scores final output will miss a tool-selection error even when the final answer happens to come out right by coincidence.
Retrieval dependency pulls in a third consideration. If retrieval determines whether an answer is any good, RAG-specific metrics like the ones Ragas provides belong in the stack no matter what else is running alongside them. Multi-agent coordination raises the bar further still: evaluating handoffs between agents requires a framework that can ingest and attribute spans across agent boundaries, which makes trace-level coverage close to mandatory rather than a nice-to-have.
Where failure attribution actually matters is its own axis, separate from all the others. Intermediate-step failure localization and aggregate outcome scoring are different products solving different problems, and an end-to-end metric will mask exactly which step broke even when it correctly reports that something did. Workflow context closes the loop: CI/CD regression testing (DeepEval's home turf), production monitoring (where Arize Phoenix and LangSmith live), and a unified development loop spanning both (MLflow's pitch) are three different primary needs, not three flavors of the same need.
Even after all that, gaps remain that the current generation of tools doesn't close. The arXiv 2604.19818 synthesis identifies safety, robustness, and trajectory-level measurement as open research problems, not solved ones, even in the strongest frameworks available today. And above the framework layer entirely sits what that same synthesis calls a governance-to-action closure gap: evaluation can say whether outcomes were good, governance can define what should be allowed, but neither one pins down where a specific obligation binds to a specific action, or how compliance gets proven after the fact. Teams building in regulated industries need to plan for that gap explicitly, because no evaluation framework on this list claims to close it.
Why the runtime where agents execute shapes what evaluation can see
An evaluation framework can only score what it can actually observe. If the environment an agent runs in doesn't expose full trace data (tool calls, memory operations, intermediate state changes), the scorer is working blind no matter how sophisticated its metrics are on paper.
The non-determinism problem discussed earlier compounds this. If variability in execution paths, tool selection, and memory retrieval means identical agent code can produce different traces run to run, the runtime itself has to be capable of reproducing those traces reliably, or evaluation results become impossible to trust from one run to the next. A scorer flagging a failure is only useful if someone can go back and confirm the failure actually happened the way the trace says it did.
Isolation matters here in a way that's easy to underrate. If agent code executes in a shared or poorly isolated environment, one agent's side effects can bleed into another run's evaluation data, and the resulting unreliability has nothing to do with whether the evaluation framework itself is well designed. The environment leaked; the framework just inherited the mess.
Long-running agents add a further requirement. Multi-step workflows that run over extended periods need a runtime that holds state across the full execution. An ephemeral environment that drops state between steps makes end-to-end trajectory evaluation impossible in principle, not just difficult in practice, because the data needed to reconstruct the trajectory simply isn't there anymore by the time evaluation runs.
That points toward a runtime layer purpose-built for these workloads: isolated, stateful, and designed from the ground up for tool-invoking, non-deterministic agents rather than retrofitted from generic containers or serverless functions built for stateless web requests. Infrastructure with sub-90ms provisioning and an open-source foundation matters here for a specific reason: it's what gives an evaluation framework, whichever one a team ends up choosing, clean, complete, and reproducible trace data to work with. Evaluation and execution turn out to be the same question asked from two different directions, and neither one gets a satisfying answer without the other holding up its end.
