LLM Benchmark Landscape for Agent Developers

Agent developers face a benchmark landscape that splits along four different capability axes: coding execution, planning, tool use, and safety, and none of them talk to each other cleanly. That fragmentation matters because the systems being built aren't chatbots. They're programs that plan across steps, call tools, hold state across a session, and (ideally) recover when something breaks mid-task. Most of the benchmarks that dominate leaderboards were built for a different job entirely: static, single-turn language understanding, where the question and answer live in the same breath. A 2025 ACL symposium survey identified this mismatch: as models evolve into agents that call tools and act inside environments, static benchmarks stop reflecting how those agents actually perform once deployed. The practical risk is straightforward. A model that leads a general leaderboard can still fail on the planning, tool-noise robustness, or long-horizon reasoning that a production agent depends on every single day. An AI Agent Index review of 30 deployed agent systems found that only 9 report capability benchmarks at all, which means the transparency gap starts before a developer even opens an evaluation report. What follows is a walk through what each benchmark family actually measures, where it stops measuring, and how to line that up against the kind of agent someone is actually trying to ship.

How the benchmark landscape is organized across capability dimensions

A 2025 ACM symposium paper offers a useful map for sorting through all this: a four-stage framework covering generation, execution, evaluation, and compliance, laid across the axes of capability, scenario, and system. Run any benchmark through that lens and four gaps appear in it again and again: task coverage that's too narrow, evaluation metrics that don't agree with each other across benchmarks, high cost to reproduce results, and a persistent misalignment with human values and safety expectations.

Translated into questions a developer can actually ask before adopting a benchmark: Does it generate tasks that resemble the agent's real domain, or a toy version of it? Does it run the agent in a live environment, or simulate the tool calls? Does it score only the final output, or does it look at the reasoning steps that got there? And does it check for safety failures and misuse, or does it only ask whether the task got done?

An IBM Research and Yale survey (arXiv:2503.16416) notes a clear shift in the field toward benchmarks that are more realistic and more complex, moving away from the simplified, static test environments that defined earlier evaluation work. Three broad families have emerged to fill that space. Coding and software-engineering benchmarks are execution-heavy, with output that's relatively easy to verify. Planning and tool-use benchmarks are process-heavy, and much harder to score without human judgment. Domain-specific and safety benchmarks are coverage-heavy and scenario-driven, built to probe edge cases rather than average performance. Each family answers a different question, and no single benchmark spans all four stages of that ACM framework reliably. That fragmentation isn't a footnote. It's the actual problem this piece is trying to work through.

What coding and software-engineering benchmarks measure, and where they break

SWE-bench Verified is the benchmark most cited in agent circles going into 2026, with headline numbers like Claude Mythos Preview at 93.9%, Claude Opus 4.7 at 87.6%, and GPT-5.3 Codex at 85%. Those numbers look clean. They are not as clean as they look, and treating them as gospel means ignoring the gaps underneath.

Roughly 19.78% of cases marked "solved" on SWE-bench Verified turn out to be semantically wrong: the test suite passes, but the actual code fix doesn't do what it's supposed to do. That's not a rounding error. It means close to a fifth of the benchmark's reported successes are passing for reasons that have little to do with correct code repair.

SWE-ABS, a version of the benchmark with adversarially strengthened test suites, makes that inflation concrete. Claude 4.5 Opus drops from 74.40% to 58.00%, a fall of 16.40 points. Gemini 3 Pro Preview drops from 74.20% to 56.80%, a fall of 17.40 points. Read those two numbers side by side and a pattern emerges: somewhere between a fifth and a quarter of "solved" problems on the original benchmark are artifacts of predictable test suites, not evidence of a model actually understanding the bug.

SWE-bench Pro, introduced in September 2025, was built as the harder answer to that problem. It draws 1,865 problems from 41 repositories, and the best models reach around 23% on the public subset, dropping to somewhere between 15% and 18% on the private subsets. That's a steep compression from the 85-to-94 range quoted above, and it says something important: the leaderboard numbers developers use for model selection are compressing a performance distribution that's much wider, and much less flattering, than it appears.

There's also the leakage question. SWE-rebench runs a decontaminated pipeline meant to catch models that may have seen benchmark problems during training, a particular risk for open-source models released in 2024 or early 2025. One model stands out here: it's the only one whose performance noticeably drops on a later 2025 subset compared to an earlier 2025 subset, a signal that some of its earlier score may reflect exposure to training data rather than genuine generalization.

For developers picking a coding-benchmark stack, LiveCodeBench and SWE-bench Verified cover coding proper, while SWE-bench Verified, WebArena, and Terminal-Bench (from Stanford and the Laude Institute, 2025) cover agentic coding capability more broadly. Terminal-Bench in particular tests agents on hard, realistic command-line tasks, which sits a lot closer to how a code-executing agent actually works day to day than repository-patch tasks alone.

Coding benchmarks answer one question well: can the model produce correct output inside a defined, verifiable environment? Planning benchmarks ask something harder. Can the model figure out what to do before it does anything at all?

Planning benchmarks and why end-to-end success scores hide the real failure modes

Most agent evaluations report a single end-to-end success rate, and that number can't tell anyone whether a failure came from bad planning or bad execution. That's the core diagnosis behind the Agent Planning Benchmark, or APB, and it's worth taking seriously because it exposes a blind spot most leaderboards don't even acknowledge.

APB runs 4,209 multimodal test cases across 22 domains and five distinct settings. It covers holistic planning, which looks at the full trajectory and tool chain an agent builds; feedback-conditioned step-wise planning, which checks how an agent adjusts mid-task; and robustness scenarios that throw in extraneous tools, broken tools, and unsolvable tasks that call for a calibrated refusal rather than a hallucinated attempt. Across 12 multimodal LLMs tested, the same weaknesses occur repeatedly in the results: long-horizon planning breaks down, tool-noise robustness is weak, refusal calibration is off, and inference-time refinement rarely happens. An end-to-end benchmark would file all of that under generic "task failure" and move on.

Why does the outcome-only framing miss this? A flawed plan can still pair with clean execution and pass. And a genuinely sound plan can fail because a tool goes down mid-task, penalizing the agent for something entirely outside its control. Neither case tells a developer anything about whether the model can actually plan.

There's a numbers problem hiding inside the failure modes, too. Analysis of multi-agent failures found that 17.14% of agent failures are step repetitions, where the agent loops on the same action, and 13.98% are mismatches between the agent's stated reasoning and its actual action. Both slip straight past a final-output check. If the task eventually completes, none of that ever appears in the score.

APB was validated against 200 ToolSandbox tasks and 200 tasks from τ²-bench, and using APB to guide refinement produced consistent gains in plan correctness and in downstream execution metrics across three representative models. That's a meaningful signal: fixing planning specifically, rather than tuning for end-to-end success, moves the needle on both.

What APB's five task types make visible, that an end-to-end score simply can't, comes down to three things. Whether the model actually understands the goal before acting on it. Whether it can rebuild a plan when a tool breaks partway through. And whether it recognizes a task is unsolvable and says so, instead of inventing a path that looks plausible but leads nowhere. The IBM Research and Yale survey backs this up from a different angle, noting a clear field-wide shift toward more realistic and complex benchmarks precisely because simplified environments fail to expose these planning limitations.

Planning benchmarks isolate one capability at a time. Most production agents need several capabilities running together: choosing the right tool, holding a conversation across turns, tracking what changed in the world since the last step. That's the terrain tool-use and multi-domain benchmarks are built to cover.

Tool-use, multi-domain, and safety benchmarks (what they cover and where each applies)

AgentBench (Liu et al., ICLR 2024) tests models as agents across eight different environments, including operating systems, databases, knowledge graphs, web shopping, and web browsing. Tasks are multi-turn, with estimated solving turns ranging from 5 to 50 depending on the environment. It's a good fit for developers who need a breadth-of-capability read across many environment types rather than depth in one.

WebArena (Zhou et al., 2023) puts agents into a self-hosted simulation covering e-commerce, social forums, collaborative code development, content management, and map navigation, across templated tasks. It scores functional correctness only: success means the agent reached the goal, regardless of the path it took to get there. That makes it a solid fit for agents that operate in realistic web environments where success is measured by whether the agent reached the goal, regardless of the path it took to get there.

GAIA (Mialon et al., 2023, ICLR 2024) runs 466 human-annotated tasks that mix text with images or files, split across three difficulty levels, from Level 1 (no tools, or at most one tool and a few steps) up to Level 3 (arbitrarily long sequences using any number of tools). It's often paired with MMLU-Pro, GPQA-Diamond, and ARC-AGI-2 when comparing frontier models head to head.

MINT (Wang et al., 2023) checks how models handle multi-turn interaction with tools and natural-language feedback: agents write and run Python code, then get GPT-4-simulated user feedback and have to adjust. It covers reasoning and QA, code generation, and decision-making, and fits developers whose agents need to adapt mid-conversation based on correction, not just execute a fixed plan.

ColBench (Zhou et al., 2025), introduced as part of the SWEET-RL paper, is a multi-turn benchmark for agents collaborating with a simulated human on backend programming and frontend design. It's built around iterative refinement, where the model suggests something, gets feedback, and revises, which maps well onto human-in-the-loop workflows.

τ-Bench (Yao et al., arXiv:2406.12045) tests tool-agent-user interaction across real-world domains, and it's one of the benchmarks APB validated against as a downstream execution check.

AppWorld (ACL 2024) benchmarks interactive coding agents inside a controllable world of apps, with particular emphasis on whether the agent can track how the world's state changes across sessions. That's a reasonable proxy for the kind of long-running, stateful execution production agents actually need to survive.

ToolEmu (Ruan et al., 2023) stands apart from the rest of this list. It covers 36 high-stakes tools and 144 test cases built around scenarios where agent misuse leads to real consequences, simulating tool execution without touching live infrastructure, and it includes an LM-based automatic safety evaluator. It's the only benchmark here designed to surface risk rather than task completion, which makes it close to mandatory for any agent that touches sensitive data or systems it could damage.

BrowseComp (OpenAI, 2025) is built specifically for browsing agents that need to retrieve and synthesize information pulled from live web sources. ForeSci runs 500 tasks across four fast-moving AI domains, testing whether an agent can make forward-looking research judgments from historical evidence, using a cutoff-aligned offline knowledge base for each task, which suits developers building research or scientific workflow agents. Webshop simulates a realistic online store for evaluating web-based shopping tasks, useful for narrowly scoped retail or procurement agents.

Line all of these up and one thing stands out: almost every tool-use and multi-domain benchmark scores final task success and stops there. ToolEmu is the exception that bothers to ask whether the agent's method of getting there was safe. That gap matters most exactly where agents have write access to systems that can't be undone with a retry.

Five structural problems that limit what any benchmark can tell a developer

Five structural problems produce nearly every benchmark discussed so far: environment drift, test flakiness, solution leakage, methodology opacity, and the gap between benchmark and production. Each one costs a developer something specific if it goes unrecognized.

Environment drift happens when the benchmark's own environment changes between runs or across time, which quietly makes scores from different model versions incomparable, even when the leaderboard presents them side by side as if they were.

Test flakiness comes from the fact that agents are stochastic. Run one and it samples a single point from a wide distribution of possible outcomes, and a confidence interval built from something like 30 samples can span a range wide enough that a "good" agent and a "mediocre" one become statistically hard to tell apart.

Solution leakage is the training-data problem: a benchmark's own test instances can end up inside a model's training set. SWE-rebench documents this concretely for open-source models released in 2024 and early 2025, and the GPT-4.1 result mentioned earlier is a direct symptom of it.

Methodology opacity is arguably the quietest of the five. Only 4 of the 30 deployed agent systems reviewed provide agent-specific system cards, and without that documentation, two models posting the same score may have been evaluated under entirely different conditions, making any cross-benchmark comparison shakier than it looks.

And then there's the benchmark-to-production gap. A benchmark task is designed. A production workload is an open distribution of whatever users actually throw at the system. Most of the failures an agent experiences in the wild live in that gap, not in the benchmark itself.

Reporting a single score from a single run treats something inherently random as if it were fixed and repeatable. No leaderboard position survives close scrutiny once confidence intervals get attached to it. And the transparency problem compounds all of this. If the methodology behind a score is opaque, there's no way to know whether two models with identical numbers were actually tested the same way. None of this means benchmarks are useless. It means each score can only carry so much weight, and a developer needs to know what other evidence has to sit alongside it before trusting it with a real decision.

How to map benchmark categories to real agentic workload types

The real question isn't which model tops the aggregate leaderboard. It's which model performs well on the specific capability dimensions the agent being built actually depends on, day to day, task by task.

For agents that write, debug, or execute code, weight SWE-bench Verified alongside its SWE-ABS delta as a sanity check, SWE-bench Pro for a harder ceiling test, and Terminal-Bench for anything CLI-heavy. For agents that plan and break tasks into steps, APB's holistic planning and broken-tool robustness scores matter far more than any end-to-end success number by itself. Agents operating across web or app environments should lean on WebArena's 812 tasks and AppWorld's cross-session state tracking. Agents holding multi-turn conversations with users should weight MINT and ColBench, both of which score adaptation to feedback rather than getting it right on the first pass.

Agents with access to high-stakes tools or sensitive data need ToolEmu's 144 safety test cases treated as required evaluation, not a nice-to-have bolted on afterward. And agents built for research or scientific workflows are better served by ForeSci's 500 temporally controlled tasks, purpose-built for judging evidence-to-decision reasoning under real uncertainty, rather than anything general-purpose.

None of these benchmarks, alone, answers the full question. That's the point this piece keeps circling back to. The fragmentation across capability dimensions isn't a flaw to be fixed by waiting for one benchmark to cover everything. It's the actual shape of the problem, and understanding that shape is what separates a defensible model-selection decision from one that just chases whatever sits on top of a leaderboard this month.