Running Agent Evaluation Benchmarks in Reproducible Environments

Benchmark scores for AI agents look precise. Percentages, leaderboards, decimal points, all the trappings of hard science. But most of them measure something closer to luck, or worse, an artifact of the test harness itself, rather than what an agent can actually do. That distinction, between a number that reflects capability and a number that reflects noise, is the entire subject of this piece.

Public benchmarks are running out of runway. By April 2026, agent evaluation had hit a point where an automated agent could score 100%, or something close to it, on seven of eight leading benchmarks without solving a single underlying task. It didn't reason through the work. It found flaws in the evaluation infrastructure and exploited those instead, which is a very different skill from writing correct code, and arguably a more impressive one, just not the one anybody was trying to measure.

The mechanism is often almost comically small. On SWE-bench Verified, one of the field's most cited coding benchmarks, editing around ten lines in a single test configuration file was enough to make all 500 tests in the suite pass. Not because the agent fixed 500 bugs. Because the test harness stopped checking properly.

The instability isn't limited to adversarial edge cases, either. Independent analysis has shown agent performance swinging from 60% accuracy on a single run down to 25% across eight consecutive runs, same agent, same model, same task set. Scaffolding choices alone, how the tools, prompts, and retry logic are wired together, can move accuracy by more than the gap between two entire model generations. So when a vendor claims a two-point improvement over last quarter's model, ask what actually moved: the model, or the plumbing around it. Most of the time, on current evidence, it's the plumbing, and that ought to change how much weight anyone puts on a leaderboard delta.

What "reproducible" means in the context of agent evaluation

Reproducible has a specific, testable meaning here. Given the same model checkpoint, the same version of the evaluation harness, and the same set of tasks, a reproducible setup returns the same score every time. Any deviation from that is telling you something about the environment, not about the agent's competence.

Reproducibility isn't a synonym for high performance, and conflating the two is one of the more common mistakes in how these numbers get read. A perfectly reproducible harness can, and should, reliably report that an agent is bad at something. The value lies in scores that mean the same thing on Tuesday as they meant on Monday. It's in scores that mean the same thing on Tuesday as they meant on Monday.

Agent evaluation runs on three layers, and all three need to be reproducible before the resulting number means anything. Final-answer evaluation checks the last message an agent produces against an expected result. It's necessary, but nowhere near sufficient on its own. Trajectory evaluation scores the full sequence, including the tool calls, the loops, and the recoveries from failure. An agent that reaches the correct final answer but makes policy-violating calls somewhere in the middle has produced a passing final answer and a failing trajectory. Final-answer scoring calls those identical. They are not identical: per-turn evaluation goes a level deeper still, scoring each turn as it happens. Per-turn evaluation goes a level deeper still, scoring each turn as it happens: jailbreak attempts, policy violations, a user's rising frustration. Evaluating only the last message hides all of that.

Treating layer one as the whole job is how a benchmark run looks green on a dashboard while the underlying trajectory looped twice and drifted off policy for three turns in the middle. The pass/fail bit hides the mess, and most published leaderboards surface exactly that bit with little else.

Six ways a non-reproducible environment silently corrupts results

Most of the damage traces back to six specific, well-documented failure points.

Container tags are mutable. Referencing a Docker image by tag rather than by digest, means a re-push of that image can silently swap out the sandbox's contents between two runs that are supposed to be identical. This is a documented anti-pattern in the container world generally, and it's exactly as dangerous in an evaluation harness as it is in a production deploy pipeline.

Seed and sampling drift is another quiet corrupter. Without pinning PYTHONHASHSEED=42, temperature to 0, and top_p to 1.0, hash-randomized data structures and ordinary sampling variation will produce different agent trajectories on inputs that are, on paper, identical.

Tightly coupled agent logic muddies comparisons across benchmarks. When a benchmark demands an agent tailored specifically to it, that agent becomes partly an artifact of the environment it was built for, which makes it close to meaningless to compare the same agent's performance across two different benchmarks.

Mocked tool APIs are the fourth, and probably the most underrated. A benchmark that asks an agent to "book a flight" produces almost no information about how that agent performs against a live, messy, rate-limited, occasionally-down production API, if the flight API driving the task is a mock. Task completion metrics only mean something when they're grounded in real tasks, not synthetic stand-ins built to be well-behaved.

If these failure modes are this well understood, why do they keep appearing in published results? Because most teams treat the environment as setup, something to configure once and forget, rather than as part of the methodology that has to be documented and versioned right alongside the agent code itself. That's the actual root cause running under all six, and it's a process failure, not a technical one.

The benchmark landscape that teams are running against in 2026

SWE-bench Verified remains the field's canonical baseline. It's a 500-instance, human-reviewed split of the larger SWE-bench, built from real GitHub issues pulled from popular Python repositories, and it scores by actually executing tests rather than relying on preference judgments from a grader model. It uses Docker for reproducible evaluation, and running the harness locally requires setting Docker up correctly. A fair number of teams introduce the tag mutability issue from the section above without realizing it. The benchmark is mature at this point, which is itself a problem: it's heavily represented in public training data now, and its maturity means top frontier scores on it warrant contamination and test-design caveats. Almost every new coding agent still gets measured against it, mostly because everyone else is too, which says more about incumbency than about the benchmark's remaining diagnostic value.

Terminal-Bench takes a different angle, dropping agents into real terminal environments to compile code, stand up servers, install dependencies, run test suites, debug failures, and patch security issues. It comes with both a task dataset and an execution harness that connects models to a sandboxed terminal, built from the ground up for reproducible runs. Terminal-Bench 2.1 adds continuous validation on top of that.

SWE-EVO asks a different question than either of those. Instead of framing the task as fixing a single issue, it asks an agent to evolve a codebase over time, which is a closer analogue to what a maintainer actually does over a sprint. Its design draws on SWE-bench's repository and execution environment foundations, with reproducibility as a stated design property rather than an afterthought.

Multi-SWE-bench pushes past the Python-only limitation that runs through most of this list by extending evaluation across multiple language ecosystems, which matters because real coding agents need to understand different package managers, build systems, and test frameworks, not just pip and pytest. That breadth also makes it a useful resource for open research groups who want reproducible evaluation without being locked into one language's tooling.

What a rigorous evaluation harness looks like in practice

The execution environment is part of the methodology, full stop. It is not setup, and treating it as setup is how the six failure modes from earlier sneak into published numbers unnoticed.

A few practices are close to mandatory at this point, and none of them are exotic. Container images should be referenced by digest, never by tag, because tags are mutable and can change what's inside the sandbox without anyone noticing. Every agent call should pin PYTHONHASHSEED=42, temperature at 0, and top_p at 1.0. And the harness itself needs version control alongside the agent's code: a change to the harness is a change to the methodology, and it should be tracked with the same discipline as a change to the model itself.

Environment reset deserves its own line item. Every task needs to start from the same clean state, not just at the boundary between full runs, but before each individual task within a run. CORE-Bench, which tasks agents with computationally reproducing published scientific results, is a useful reference point for what rigorous per-task isolation looks like in practice.

One structural requirement sits at the center of all of it: a clean separation between the benchmark infrastructure and the agent code being tested. That separation is what lets a team swap in a different agent without quietly changing what's actually being measured. Skipping it means every new agent version risks becoming, in effect, a new benchmark, which defeats the purpose of having a benchmark.

Isolation needs in agent benchmark execution environments

Agents running benchmark tasks are executing real code. They install dependencies, compile projects, run test suites, call external APIs, and modify filesystems. This code is untrusted and AI-generated, and it stays untrusted running inside a benchmark harness or a production pipeline alike. Treating a benchmark sandbox as somehow lower-stakes than a production one causes a lot of the noisy results discussed above, as seen in those results themselves.

Standard containers share the host kernel with everything else running on that machine. Security guidance in this space is consistent: containers don't offer as clear a security boundary as virtual machines do. That guidance was written with production workloads in mind, but it applies just as directly to an evaluation harness running dozens of agent tasks back to back.

The isolation model chosen for a harness has consequences for reproducibility, not just for security. Shared-kernel containers can let one task's side effects leak into the next task's run, and this appears in results as agent variance when it is actually an environment failure wearing a disguise. MicroVMs, the kind built with Firecracker or Kata Containers, give each task a dedicated kernel and the strongest isolation on offer, at the cost of a slower startup. gVisor sits in between, providing a stronger isolation boundary than standard containers without spinning up a full virtual machine. V8 isolates go the other direction entirely, offering startup times measured in microseconds for JavaScript functions, trading language flexibility for a smaller attack surface.

For the kind of task nearly every serious coding benchmark demands, installing packages, running build tools, working against a full Linux filesystem, microVMs are the only option that gives the fidelity of a real machine alongside hardware-level isolation. Anything lighter is a compromise dressed up as an optimization. A benchmark tuned purely for fast iteration will reach for lighter isolation almost by default, and that choice quietly becomes a methodology decision, regardless of whether anyone frames it that way.

Sandbox platforms that support reproducible agent evaluation at scale

Judging a sandbox platform against everything above means checking isolation model, session duration limits, startup speed, compliance posture, whether it supports bring-your-own-cloud deployment, and Docker compatibility. Not every platform on the market treats all of that as a first-class concern, and it shows the moment a team pushes past a toy workload.

Northflank uses Kata Containers and gVisor for isolation and reports processing over 2 million isolated workloads a month. Its benchmarked startup speed is a 97ms median time-to-interactive with a 100% success rate, and it which matters for teams that need evaluation infrastructure with strong isolation guarantees at scale.

Vercel Sandbox runs on isolated Firecracker microVMs, built specifically for safely running untrusted or AI-generated code. Its main appeal for evaluation work is structural rather than a specific benchmark number: there's no separate infrastructure to stand up and manage, which matters for teams that want their evaluation environment to mirror the same stack they deploy to in production, instead of maintaining two systems that quietly drift apart from each other over time.

Cloudflare Sandboxes reached general availability on April 13, 2026.

A benchmark score is only as trustworthy as the environment that produced it. That's been true the whole way through this piece: the topic was a mutable Docker tag, a mocked flight API, or a shared kernel leaking state between tasks. In 2026, the environment has stopped being a footnote. It's the methodology.