Memory Checkpointing and Restore for Sandboxed Workloads

Memory checkpointing and restore for sandboxed workloads is an OS-level problem, and saving a transcript of what an agent said and planned to do is not the same as capturing the state it would need to resume execution. Application-level persistence (framework history, file state, conversational context) records what the agent said and intended, but it does not record what the operating system actually did in response. An agent that remembers deciding to install a package is not the same as a sandbox that still has that package installed, correctly linked, with the right environment variables set. Crab, the checkpoint-and-restore runtime for agent sandboxes built by Wu and colleagues, draws this line with precision: application-level methods "preserve conversational or file state cheaply but miss OS-side effects such as installed packages, spawned processes, and files modified by shell commands." The consequence is visible at the worst possible moment: if an agent crashes after running pip install, rm, or sed, restoring from chat history alone means replaying those commands, which can duplicate side effects, re-send an email, or re-charge a payment that already went through.

The agent-OS semantic gap and why it makes naive checkpointing expensive

The reason checkpoint-and-restore for agent sandboxes is hard to do cheaply traces back to a structural blind spot: neither the agent layer nor the OS layer alone holds enough information to checkpoint efficiently and correctly. Crab names this disconnect the "agent-OS semantic gap," and treats it as the structural cause behind two distinct failure modes that teams building agent infrastructure keep running into. One might argue that the fix is simply to checkpoint more often and accept the cost, but Crab's own measurement cuts against that instinct: more than three-quarters of agent turns produce no recovery-relevant OS state. That sparsity means naive per-turn checkpointing is almost always unnecessary work, and existing tools have no way to tell which turns actually require it.

The three requirements of a correct checkpoint

Before weighing one checkpoint mechanism against another, it helps to fix what "correct" even means in this setting, because the bar for agent sandboxes sits higher than it does for general serverless functions or developer tooling. Crab articulates three requirements that any checkpoint-and-restore system for agent sandboxes has to satisfy together. These three pull against each other in practice. A structural detail makes satisfying all three requirements jointly harder than it looks: sandbox state has two tightly coupled dimensions, durable filesystem state and ephemeral process state (memory, open file descriptors, execution context), and DeltaBox notes that managing these two dimensions separately leads to state divergence or lost context. Any system that treats files and process memory as two unrelated bookkeeping problems risks producing a checkpoint that is internally inconsistent, restoring a filesystem that no longer matches the process that's supposed to read from it.

How full-state duplication approaches work

Today's tooling is dominated by two production-grade checkpoint-and-restore patterns, and both were built for workloads that look different from high-frequency agent execution. The second pattern is event-history replay, exemplified by Temporal: it records every state transition as an immutable entry in an append-only log, and it rebuilds prior state when it replays that log on recovery. Replay latency grows with history depth, and the approach does not capture OS-side effects that sit outside the workflow abstraction.

Both patterns share the same underlying bottleneck: full-state duplication or replay of state that leaves the changed portion unisolated. DeltaBox frames this as two concrete, separable challenges. CRIU, the standard tool for process-state checkpointing on Linux, serializes process memory sequentially, so its restore time scales with process size. Fork latency at that scale bounds training throughput directly, so no amount of parallel hardware can fix a bottleneck that sits in how long each individual sandbox takes to restore.

Delta-state checkpointing: capturing only what changed between turns

The insight that breaks this bottleneck is almost embarrassingly simple once stated: consecutive sandbox states in agent workloads differ only slightly, so a system that tracks the change rather than the whole state can be both faster and no less correct. DeltaBox formalizes this as the DeltaState abstraction, an OS-level primitive for change-based, transactional checkpoint and rollback, realized through two mechanisms designed together. That pairing matters: the earlier point about filesystem and process state needing joint treatment is what DeltaFS and DeltaCR are built to satisfy.

DeltaFS handles the filesystem side by organizing file state in layers, using a new standalone filesystem that extends Linux overlayfs. At each checkpoint, DeltaFS freezes the current writable layer and inserts a new one on top of it. DeltaCR handles the process side with a comparable move: at every checkpoint it performs both an incremental CRIU dump, for durability, and a template-creating fork(), for low-millisecond restore. Rollback accelerates by forking directly from a frozen template process, bypassing the traditional CRIU restore pipeline. An asynchronous hot-page warm-up step, called async-warm, pre-pays copy-on-write faults on anonymous writable pages after the fork completes, hiding that cost from the agent's critical path.

The measured result: DeltaBox hides roughly 10.83 milliseconds of checkpoint work under LLM inference time, time the agent is already spending waiting on a model response, and achieves rollback in around 1.86 milliseconds, comfortably inside the latency budget of a single reasoning step. What that buys in practice is room to explore. Agents can search substantially more nodes inside a fixed time budget, and search strategies that depend on cheap, frequent rollback to explore many candidate branches become viable at a scale that was previously throttled by the cost of checkpointing itself.

Semantics-aware checkpointing: aligning C/R with agent turn boundaries

Delta-state checkpointing answers one half of the problem by making each individual checkpoint cheaper. Semantics-aware checkpointing addresses how many checkpoints get taken in the first place, using the agent's own turn context to decide which OS-level changes are worth preserving. Crab operationalizes this directly against the semantic gap described earlier: you get a transparent host-side runtime that bridges agent and OS without needing modifications to either the agent framework or the underlying C/R backend.

Crab is built from three components. An eBPF-based inspector observes the OS-visible effects of each agent turn once that turn completes, and classifies whether those effects are recovery-relevant, so you get per-turn granularity without any instrumentation on the agent side. A coordinator aligns checkpoints with turn boundaries rather than arbitrary time intervals, and overlaps checkpoint work with LLM wait time: the natural pause between a tool call finishing and the next reasoning step beginning becomes free capacity to absorb checkpoint cost, a structural property of agent workloads that general-purpose checkpoint systems have no reason to exploit because they were never built around the rhythm of LLM inference. A host-scoped engine schedules checkpoint I/O across every sandbox co-located on a shared host, so no single burst of checkpoint traffic can saturate shared storage.

The measured outcome on shell-intensive and code-repair workloads is stark. Recovery correctness rises from 8 percent under a chat-only baseline to 100 percent under Crab, checkpoint traffic drops by up to 87 percent, and end-to-end overhead stays within 1.9 percent of fault-free execution time. That jump from 8 to 100 illustrates what the semantic gap costs in practice: naive application-level recovery fails on nine out of ten recovery attempts on realistic agent workloads because it is recovering the wrong layer of state.

Computer-use and GUI agents: checkpointing beyond the shell

Shell-based agents checkpoint filesystem and process state, but agents that operate inside live graphical environments accumulate more than that: GUI state, open applications, authenticated sessions, and credentials, all of which need to be captured for a restore to be safe. A computer-use agent that cannot roll back cleanly can modify files, close applications mid-task, drain stored credentials, or corrupt an authenticated session in ways that are difficult or impossible for a human to reverse afterward. Rollback here is a safety mechanism before it is a performance feature.

TClone takes on the tension this creates directly: agents need isolation and rollback to avoid damaging user state, but they also need fast branching to support speculative execution and parallel search across the space of possible GUI actions. The pattern has started appearing at the platform level too: Cloudflare Sandboxes, which reached general availability in April 2026, expose a programmatic snapshot API that lets operators fork sandboxes from the same filesystem snapshot to run parallel agent instances.

Long-running agents and preemption

The agents running in production today don't fit the ephemeral, stateless sandbox model. They run for hours, pause mid-task waiting on human approvals, need to survive crashes without losing hundreds of turns of progress, and often sit idle, billing nothing, while they wait for a response. Three scenarios make stateful checkpoint-and-restore the mechanism that makes these workloads possible, not a performance add-on layered on afterward. Fault tolerance on long-horizon tasks means an agent running hundreds of turns cannot afford to restart from the beginning after a crash: every prior tool call needs to be restorable rather than replayed, for the same duplication reasons discussed earlier. Human-in-the-loop approval gates present a third case: an agent that pauses for an approval arriving the next morning has to preserve not just its conversational context but its entire OS state, including open files, running processes, and partially completed tool chains.

Pulumi's experience illustrates what happens when a team runs into these requirements directly. Pulumi's answer was to make checkpointing asynchronous background work that happens after the response has already returned to the user, so the slower checkpoint mechanism never blocks the interaction itself. That choice is a useful data point for any team weighing checkpoint latency against checkpoint correctness: sometimes the fix isn't a faster primitive, but a different place in the execution timeline to put the cost.

Security risks specific to checkpoint state

Checkpoint files are serialized execution state, not inert performance artifacts, and a system that restores from a checkpoint which has been tampered with resumes execution with falsified prior context baked in. A 2026 paper names this threat class "semantic rollback attacks," describing an adversary who exploits the gap between an agent's rolled-back local state and external state that was never rolled back. Neither attack has a real analog in traditional checkpoint-and-restore systems, because traditional workloads don't have an agent layer making consequential decisions based on a belief about what it previously did.

What follows is that checkpoint storage needs the same isolation guarantees as the sandbox runtime itself: encryption at rest, integrity verification at restore time, and scoped credentials so that no sandbox can read another tenant's checkpoint state. MicroVMs such as Firecracker and Kata Containers give you a strong hardware-level isolation boundary against untrusted code, and they prevent shared-kernel exploit classes, but microarchitectural and side-channel attacks may still need additional host-level mitigations on top of that boundary. Checkpoint state for microVM-backed sandboxes only inherits that isolation if the checkpoint pipeline itself is isolated to the same standard, since a checkpoint pipeline with weaker guarantees than the sandbox it serves becomes the easier target. Compliance frameworks add one more requirement that checkpointing is uniquely positioned to satisfy: SOC 2 Type II says nothing about whether agentic actions produce an audit trail that satisfies an enterprise regulator, and a durable, tamper-evident checkpoint chain is the mechanism that closes that particular gap.

What these techniques require from the sandbox platform

Delta-state and semantics-aware checkpointing aren't features that snap onto a generic container platform after the fact. Each one requires the sandbox runtime underneath to be built around it from the start. DeltaFS needs the ability to dynamically freeze and insert overlayfs layers at the moment of checkpoint, a capability standard container runtimes don't expose without kernel-level modification. DeltaCR needs the ability to fork from a frozen template process and run incremental CRIU dumps, which requires host-level access that tenanted cloud containers typically don't grant their users. Crab's eBPF inspector needs to interpose at the host layer, sitting between the agent framework and the kernel, which requires the platform itself to own the execution boundary. And Crab's host-scoped scheduling needs the platform to coordinate checkpoint I/O across every sandbox sharing a host, something a platform that treats each sandbox as fully independent has no way to do.

What ties these requirements together is that statefulness has to be a design decision made at the platform level, not a retrofit added once customers start asking for it. That design decision is what lets agents pause, resume from a checkpoint, and pick up long-running work where they left off. No amount of clever checkpoint tooling bolted onto an ephemeral sandbox fixes the underlying limitation if the sandbox itself was never built to persist. Docker-native compatibility still matters here for portability's sake: teams running agent workloads in standard containers should be able to bring their existing images onto a platform that adds delta-state checkpoint-and-restore without forcing a rewrite of their stack. And for teams operating under data sovereignty requirements, customer-managed compute is a direct consequence of everything in the previous section, since checkpoint data contains serialized execution state that may include credentials, intermediate tool outputs, and other sensitive context that cannot simply be handed to a shared multi-tenant object store outside their control.