Snapshot and Restore for GPU Agent Eval Environments

Snapshot and Restore for GPU Agent Eval Environments.

Why GPU agent eval environments are harder to reproduce than they look

The decisive question for AI infrastructure in 2026 is not whether agents can act autonomously; it's whether the runtimes underneath them can keep pace with what autonomy actually demands AI 2026: Infrastructure, Agents, and the Next Cloud-Native Shift | Jimmy Song. Eval infrastructure has become a bottleneck rather than a footnote, which is why the distinction matters before getting into snapshot and restore mechanics. Static benchmarks like BLEU and ROUGE never had to grapple with this problem: they scored fixed outputs against fixed references, no live environment required. Agent evaluation looks nothing like that anymore. Behavioral metrics, things like intervention rates, rollback frequency, and cost impact, require the agent to actually run, which means a live sandbox has to exist for every single task.

Look at what the major eval suites actually demand and the scale of the problem gets concrete fast. SWE-bench Verified runs 500 tasks, and each one needs its own live Docker sandbox with a checked-out repository and a deterministic test runner sitting behind it AI Agent Benchmarking Infrastructure on GPU Cloud: Run SWE-bench, GAIA, Terminal-Bench, and OSWorld at Scale (2026 Guide) | Spheron Blog. GAIA pushes further, with 466 multi-step tasks that lean on web tools, file parsers, and calculators, all of which have to be provisioned and working before the agent takes its first action AI Agent Benchmarking Infrastructure on GPU Cloud: Run SWE-bench, GAIA, Terminal-Bench, and OSWorld at Scale (2026 Guide) | Spheron Blog. OSWorld is the heaviest of the bunch: 369 desktop GUI tasks that need QEMU/KVM virtualization just to render a usable interface, making it the most infrastructure-intensive of the major suites AI Agent Benchmarking Infrastructure on GPU Cloud: Run SWE-bench, GAIA, Terminal-Bench, and OSWorld at Scale (2026 Guide) | Spheron Blog. Terminal-Bench, BrowseComp, and tau-bench round out the landscape with their own sandbox demands, roughly 89, 1,266, and 165 tasks respectively.

Multiply that out and the compute cost becomes obvious. The benchmark didn't get easier. The compute architecture changed underneath it. They're symptoms of the same underlying gap: eval infrastructure that treats every run as a brand-new machine instead of a known, capturable state. GPU utilization in many AI environments remains below 30–40%, per Jimmy Song's analysis (waste that compounds when every eval run cold-starts from scratch) AI 2026: Infrastructure, Agents, and the Next Cloud-Native Shift | Jimmy Song.

What snapshot and restore means in a GPU eval context

Strip away the jargon and snapshot and restore is a simple idea: capture the complete state of a running sandbox, its memory, filesystem, GPU memory, and process state, at a known checkpoint, so that state can be restored identically later or forked into as many parallel copies as needed. The "identically" part is the whole point. A snapshot isn't a rough approximation of where things stood; it's the exact configuration, byte for byte, that existed at the moment of capture.

Why does that matter more in eval than in, say, a typical web application? Consider an agent ten turns into a Python task AI 2026: Infrastructure, Agents, and the Next Cloud-Native Shift | Jimmy Song. Losing that state means re-doing all of it. None of that reconstructs itself quickly. It's precisely this expense that makes GPU memory snapshotting worth the engineering effort, rather than a nice-to-have bolted onto an existing container platform.

In production, two backends dominate how this actually gets implemented: CRIU handles process state, OpenZFS handles filesystem state. The Crab research system, published on arxiv, documents this combination as representative of what production teams had already been assembling on an ad hoc basis. A snapshot is not the same thing as a container image. An image is a reproducible build recipe, a set of instructions for constructing an environment from scratch. A snapshot captures live, mid-execution runtime state, the actual condition of a process that's already been running and doing work. Both have their place, and confusing them is a common way eval infrastructure quietly breaks.

The market has settled on a handful of architectural approaches to persistence. Some platforms keep the workspace live: a block-storage volume that stays mounted and ready, with nothing copied out on pause and nothing restored on resume, an approach that leans on sustained IOPS to keep things responsive. Others take full standby snapshots that include running processes, so that reconnecting to a paused sandbox instantly restores whatever was executing at pause time. Neither approach is universally correct; the right choice depends on how often sandboxes pause, how large the working state is, and how much resume latency an eval pipeline can tolerate. GPU eval environments add model weights loaded into GPU memory, CUDA context, and warm KV-cache, state that is expensive to reconstruct.

The cold-start tax that makes naive parallel evals prohibitively expensive

The arithmetic makes snapshot infrastructure non-negotiable rather than a convenience. That number looks small in isolation. It stops looking small once it's multiplied by 500 tasks in a naive SWE-bench pipeline, each one re-paying that same setup tax, and then multiplied again across hundreds of parallel workers. The aggregate waste is not a rounding error at that point; it's a meaningful fraction of the total compute bill.

GPU warm-up adds another layer: loading model weights, initializing CUDA context, and warming KV-cache are not millisecond operations. That's exactly why GPU memory snapshots exist: the cost of rebuilding that state from nothing is high enough to justify the engineering complexity of preserving it instead. Standard Kubernetes pod scheduling, for what it's worth, was never built with this problem in mind. It works reasonably well for git-push web services that spin up once and serve traffic for hours. It struggles with agent turns that need fast environment handoff, because vanilla Kubernetes doesn't bundle any snapshot-capable runtime out of the box.

GPU utilization below 30–40% is a symptom of GPUs sitting idle during cold-start, setup, and teardown phases that snapshots would eliminate. That reframes the economics of eval pipelines in a useful way. Spot instances offer real savings for sandbox deployment, but they can be preempted on short notice, and without checkpoint and restore, a preemption means throwing away every turn of work that came before it and starting the episode over. With a checkpoint in place, the same preemption is closer to a shrug: the agent resumes from exactly where it paused.

Put plainly, the cost of reproducibility doesn't disappear if a team skips building snapshot infrastructure. It just gets paid differently, either upfront in engineering effort, or repeatedly, in wasted GPU-hours and the engineer time spent debugging runs that never should have needed to restart. Cold Firecracker boot baseline: 200–500ms for environment setup alone, before the agent does any work.

How snapshot/restore makes eval results reproducible rather than merely repeatable

Eval teams conflate repeatable and reproducible constantly, and the distinction matters. Unlike a unit test, which evaluates a pure function against a fixed input, an agent eval accumulates state across turns: installed packages, written files, tool outputs layered one on top of the next. Two runs that both start from "the same image" can still diverge if the environment setup underneath them isn't byte-identical. Nominally equivalent isn't identical, and in agent eval, the gap between those two things is where irreproducible results come from.

SWE-bench is the clearest illustration of why this matters. Its sandbox is explicitly designed to be deterministic: no internet access during the eval, no external state to contaminate results, a pass or fail verdict decided by pytest. But that determinism is only as trustworthy as the repo checkout and dependency state feeding into it. A snapshot taken at a verified, known-good post-setup point removes the ambiguity entirely, because every subsequent run forks from that exact configuration rather than a fresh install that might resolve a different dependency version on a different day.

What counts as "identical," though, shifts depending on the kind of agent being evaluated, and this is where a lot of eval harnesses cut corners. A code agent's meaningful state includes the branch, the file tree hash, which files changed, lockfiles, and test output. A browser agent's meaningful state is a different animal entirely, comprising the URL, form state, the extracted page packet, and the click target the agent intended to hit next. A document agent cares about document IDs, the prior chunk boundaries, which embedding model version produced them, and source hashes. Treating all of these as "just capture the filesystem" misses the point. The Crab system's framing calls this being semantics-aware: the snapshot has to capture not just raw bytes but the meaningful state the agent was actually acting on, a more demanding standard than a generic disk image satisfies. That distinction is also what makes safe rollback possible. If an agent takes a catastrophic or adversarially induced action mid-eval, an operator can restore the sandbox to the last known-good checkpoint and re-run from there, rather than discarding the whole episode and starting over.

Forking from a snapshot to run parallel eval branches at scale

Once a known-good snapshot exists, the natural next move is to fork it. Instead of cold-starting N sandboxes independently, each paying its own setup tax, a team spins up one warm template, snapshots it once, and forks N identical children from that single snapshot. Every child inherits byte-identical starting state, no exceptions.

The performance numbers behind this deserve close attention. That's sub-linear growth, dominated by the cost of copying kernel page tables rather than copying actual memory contents DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox C…. Children inherit the parent's pages copy-on-write, so per-child resident memory stays around 11MB regardless of how large the parent's total state is DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox C…. Why does copy-on-write matter so much specifically for GPU eval? Because the model weights sitting in GPU memory, the loaded environment, don't get duplicated 64 times over DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox C…. Only the state that actually diverges between children gets allocated fresh.

Tree-based reinforcement learning post-training is the workload that originally motivated this pattern: algorithms that branch from intermediate states, spawning N parallel children from a single warm template at each training step. The same fork-from-snapshot logic that makes that efficient for RL training turns out to be exactly what branching eval runs from a shared checkpoint needs. Salesforce's SFR-RL system gives a sense of what this looks like at production scale: up to 4,000 concurrent stateful code-execution environments running on a single machine, with automatic failure detection, weight restoration, and re-queuing of in-flight work, all without a human stepping in to intervene. Checkpoint and restore is the mechanism that produces that scale, even when it's not the headline feature being marketed.

The scope of modern eval suites makes this kind of parallelism close to mandatory rather than optional. GLM-5's evaluation setup, per Hugging Face's overview of RL environments, spans more than 10,000 verifiable environments across thousands of repositories, covering nine programming languages. No serial pipeline touches that volume in any reasonable timeframe. Practically, this pairs well with a queue-based harness design: stateless rollout workers pull tasks from a shared queue, and if a worker gets preempted mid-task, the task drops back into the queue and the next available worker picks it up by restoring from the same snapshot and continuing on. Nothing about the eval result changes because of where the work happened to execute.

Resuming long-running agents without losing eval progress

Long-horizon eval workloads are where the cost of skipping checkpoint infrastructure turns from theoretical to painful. The eval infrastructure has to preserve both the environment's actual state and the agent's full interaction history, not just one or the other.

What happens without a checkpoint when something goes wrong mid-episode? Every prior turn gets thrown away and re-executed from scratch. At GPU rates, a long-horizon agent run interrupted mid-episode that forces re-execution of all prior turns is not a trivial cost, and it's exactly the failure mode SFR-RL's system exists to address, since a single engine crash can stall an entire batch. Some infrastructure teams frame the unit of work as resumable microVM sandboxes built to preserve long-lived model and environment state, where the sandbox is a machine that gets checkpointed through a long attempt and resumed, never simply discarded when something interrupts it.

Spot instance economics make the same point from a different angle. GPU spot capacity can be pulled back on short notice, and a preemption without a checkpoint in place means the work is gone. With a checkpoint, the agent picks back up from exactly where it stopped. Long-horizon eval workloads: agent benchmarks like GAIA (466 tasks, multi-step) and OSWorld (369 tasks, sequential GUI control) involve agents that accumulate meaningful state across many turns. In SWE tasks, an agent inspects files, executes commands, observes tool outputs, and iteratively refines, producing context explosion AI Agent Benchmarking Infrastructure on GPU Cloud: Run SWE-bench, GAIA, Terminal-Bench, and OSWorld at Scale (2026 Guide) | Spheron Blog. Context length grows rapidly across turns, often reaching tens of thousands of tokens; the eval infrastructure must preserve both the environment state and the agent's interaction history. Firecracker snapshot-restore in production: dropping from a 200–500ms cold boot to 5–30ms per sandbox means the resume cost for a checkpointed long-running agent is negligible relative to re-execution cost.

GPU passthrough and isolation: what the sandbox layer must provide for eval workloads

None of the snapshot machinery above matters if the sandbox underneath it isn't properly isolated, and eval workloads carry a specific version of this risk: they're routinely running unverified, AI-generated code that must not be able to touch the host machine or bleed into a neighboring eval task. That's not a hypothetical concern. Veracode's benchmark found that 45 percent of generated code samples failed security tests involving OWASP Top 10 vulnerability categories. Nearly half.

Documented incidents back this up with real consequences: AI coding agents have deleted local user files and, in worse cases, production databases. The Snowflake Cortex Code CLI vulnerability, disclosed by PromptArmor with coordinated public disclosure on March 16, 2026 and patched in version 1.0.25 on February 28, 2026, showed how indirect prompt injection could bypass sandbox mode entirely and reach cached credentials. A snapshot of a compromised sandbox is just a preserved copy of compromised state, which turns a snapshot into a liability rather than an asset. Isolation has to be established before the snapshot is taken, or the checkpoint isn't a genuine known-good state at all.

The isolation options available today sit on a rough hierarchy. MicroVMs, Firecracker and Kata Containers being the notable examples, give each workload its own dedicated kernel, which is the strongest isolation guarantee available and the recommended choice for genuinely untrusted AI-generated code. Standard containers share the host kernel, and that shared kernel produces escape vectors: CVE-2024-21626, nicknamed "Leaky Vessels," demonstrated a file descriptor leak that let a container escape and reach the host filesystem Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform. V8 isolates are at the fast end of the spectrum, appropriate for JavaScript-scoped workloads with near-instant startup, but they're not built for arbitrary GPU workloads.

A few platforms illustrate how these tradeoffs occur in practice. VM-based development environments built around terminal sessions, repo mounting, and SDK-controlled execution let teams define reproducible environment blueprints and then snapshot the disk state of a running instance directly. Vercel Sandbox, which reached general availability on January 30, 2026, runs on Firecracker microVM clusters through an internal platform called Hive, and ships automatic filesystem snapshot on stop as an on-by-default behavior, with full persistence reaching GA on May 26, 2026. That's a meaningful signal for where the industry is heading: snapshot capability shipping as a default expectation, not a premium feature bolted on for advanced users.

Choosing an architecture: what to evaluate when building GPU eval snapshot infrastructure

So what should a team actually check before committing to a snapshot architecture for GPU eval work? A short list of variables does most of the real work here, functioning as a checklist rather than abstract considerations.

Snapshot scope comes first: does the platform actually capture GPU memory state, or does it stop at filesystem and process state? Whether GPU warm-up cost gets eliminated or just deferred depends on that single distinction.

Fork semantics matter just as much, particularly for teams running large parallel eval suites: can the platform spin up many identical children from a single snapshot using copy-on-write memory, the way DeltaBox demonstrates at sub-linear latency growth, or does every forked sandbox pay its own independent setup cost DeltaBox: Scaling Stateful AI Agents with Millisecond-Level Sandbox C…? Isolation strength has to be evaluated honestly against what the eval workload actually runs. Untrusted, AI-generated code pushed through a shared-kernel container carries a different risk profile than the same code running inside a microVM with its own dedicated kernel, and a compromised isolation layer defeats any amount of snapshot sophistication Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platform.

None of these questions has a universally correct answer, but they're no longer optional to ask. As eval suites scale into the thousands of environments and agent runs stretch across tens of thousands of tokens of accumulated context, the infrastructure choices made at the snapshot layer end up deciding whether an eval pipeline runs at a defensible cost or bleeds GPU-hours on setup that never needed to happen twice. Resume latency: Firecracker snapshot-restore brings cold-boot time from 200–500ms down to 5–30ms, so what does the platform actually achieve in production?.