Why Sandbox Cold-Start Time Is a Product Problem, Not Just Ops

Sandbox cold-start time affects how quickly an agent responds, whether it finishes a task, and how much users trust it to do so again. It is therefore a product requirement. Treating it otherwise leaves decisions about latency, state, and isolation in a backlog governed by cost and throughput that overlooks the experience those decisions create.

Why cold-start latency belongs in the product conversation, not the ops backlog

Infrastructure teams optimize for throughput and cost, while product teams answer for the experience users feel. Sandbox provisioning sits between those responsibilities, so an interaction can suffer even while each team works toward its own goals. The gap is organizational before it is technical.

For a tool call that requires a sandbox, code execution, file I/O, or a browsing action cannot proceed until that environment is live. The delay propagates into the user-facing interaction. It does not stay tucked away in an infrastructure log. If the product appears stalled, what justification is there for treating the cause as separate from product quality?

What actually happens during a cold start and why the delay is structural

A cold start creates an isolated execution boundary from scratch. That takes work, and the isolation model determines what work is necessary. Even with better hardware, building the boundary takes work. The architecture thus puts a lower limit on latency before the agent’s own logic starts to run.

Shared-kernel containers, including Docker and gVisor, can have cold starts stretching to several seconds, while a kernel exploit can escape the shared boundary. Firecracker microVMs instead use hardware-level isolation through KVM, with cold starts under 125 milliseconds. That is faster than traditional VMs but still adds a pause to real-time tool execution.

Purpose-built virtual machine monitors take a different approach. Tencent Cloud’s CubeSandbox/CubeVM uses custom KVM implementations, pre-created resource pools, and copy-on-write snapshot cloning for agent workloads. A single cold start takes well under 100ms, but P99 under concurrent load reaches the low hundreds of milliseconds. Its architecture also uses isolated action forks, a technique independently described in AOSpec.

These costs become more consequential when execution is serialized. AOSpec shows that model generation and environment execution ordinarily proceed in sequence through action, tool execution, and observation. As model decoding accelerates, tool execution occupies a growing share of end-to-end latency. A faster model can therefore expose, rather than eliminate, the runtime bottleneck.

How sandbox latency surfaces as user-facing product degradation

Users experience the whole task. In a multi-step agent loop, a cold start costing hundreds of milliseconds per tool call can compound across dozens of calls. Each wait delays the observation needed for the next action. What looks modest on a provisioning chart can become sluggishness or a mid-task stall in the product.

AOSpec demonstrates that this serialized waiting is measurable. On Terminal-Bench, overlapping model generation and tool execution reduces mean end-to-end latency and cuts P99 latency by up to 42.8%. The observation model also works on another software-engineering benchmark. These results expose time that users otherwise spend waiting for the agent loop to advance, rather than merely establish a faster benchmark result.

A median describes the middle interaction, but an unpredictable slow response can make the product seem unreliable even when most responses are fast. The user cannot plan around an unexplained pause. P99 therefore captures a dimension of quality that an attractive typical response time can obscure.

The consequence can also extend beyond perceived speed. If provisioning adds unacceptable latency at a branch point, an agent may time out or leave work incomplete. How useful is a quick first reply if the environment cannot support the steps needed to finish?

The cold-start safety gap: a second dimension of product risk that warm pools do not fix

Speed is not the only cold-start risk. A fresh agent session can also be more vulnerable to unsafe behavior, introducing a failure mode that fast provisioning alone does not address. Runtime readiness and behavioral readiness are different conditions. A warm pool can supply the former without supplying the latter.

The University of California, San Diego’s cold-start safety gap paper introduces SODA, or Safety Over Depth for Agents, to evaluate threats at controlled conversation depths. Across Llama, Qwen3, Qwen3.5, and Gemma, it finds that safety improves by 9–52 percentage points as preceding regular agentic tasks increase from zero to twenty. The preceding work is ordinary tool use. The paper’s representation analysis describes hidden states shifting across a linear safety boundary toward a safer region as those tasks accumulate.

This places a product decision at the point of greatest vulnerability. A fresh agent immediately given a sensitive or adversarially crafted request has the least preceding context when it encounters the threat. The finding does not make every sandbox restart a behavioral reset, because execution state and conversation history are distinct. It does mean that session initialization cannot be treated solely as a speed problem or left entirely to model alignment.

The paper recommends that the agent complete a few regular agentic tasks before it faces safety-critical requests. Yet maintaining that warmth costs compute and raises state-hygiene questions in a multi-tenant product. Reusing a prior tenant’s state creates a security and compliance risk, while clearing the agent context removes the history responsible for the warm-up benefit.

Why stateful sandboxes resolve the tension that ephemeral ones cannot

A stateful design gives the agent its own continuing environment, keeping warmed state within tenant boundaries. Subsequent calls can resume without recreating the sandbox, while the agent retains its own behavioral context. Continuity belongs to the agent. This separates the benefit of persistence from the risk of cross-tenant contamination.

Sandbox identity, process state, and behavioral context persist even when compute is dormant, allowing work to resume. For tree search and reinforcement learning, that continuity must also support rapid exploration and reversal. Persistence is useful only if managing it does not consume the task’s time budget.

DeltaBox provides concrete evidence that fine-grained state management can be fast. It describes existing mechanisms, including Docker commits, git stash, and VM-level snapshots, as duplicating entire states at checkpoints and taking hundreds of milliseconds to seconds per operation. Its diff-based approach uses the small changes from one agent state to the next. Checkpoint takes approximately 10.83ms and rollback approximately 1.86ms, allowing more nodes to be explored within a fixed time budget.

DeltaFS freezes the writable filesystem layer and inserts a new one, making updates copy-on-write and rollback a layer switch. DeltaCR restores process state by forking from a frozen template process backed by a CRIU dump. These mechanisms address files and running processes together.

But does persistence just keep data that should be gone? An unmonitored sandbox can indeed accumulate inappropriate state. The relevant distinction is controlled versus uncontrolled persistence. Strong isolation, continuous monitoring, and a persistent audit trail make identity and state continuity traceable rather than leaving longevity itself to stand in for risk.

Overlapping model generation and tool execution: speculation as a latency architecture

Reducing exposed waiting need not mean shortening every operation. AOSpec, from Imperial College London, overlaps model generation and tool execution by starting work that is likely to come next before its usual turn. The aim is to hide execution behind concurrent work.

Expected Value Decoding, or EVD, selects observation candidates using both probability and estimated tool execution time. It optimizes expected time hidden. This steers speculation toward the few slow calls that take up most of the total tool time. A likely prediction has limited value if it saves almost no waiting.

Some observations, however, become knowable only through execution. For those, AOSpec launches speculative actions in isolated sandbox forks that contain their effects. An incorrect speculation can then be discarded without contaminating committed state. Joint Action–State Verification, or JASV, checks both the speculative action and its origin state against committed execution before reuse, enabling longer lookahead without predicting every intervening action correctly.

Across Terminal-Bench’s four harnesses and five models, AOSpec reduces mean end-to-end latency by 11.8–32.5%. Its gains increase as decoding accelerates, and its observation model transfers to a software engineering benchmark without retraining. The infrastructure dependency is decisive. Isolated forks must be cheap to provision, verify, and discard, so speculation relies on fast, stateful sandbox execution.

Security and compliance as the enterprise product gate that latency optimization must not bypass

Fast execution does not make an enterprise product deployable. If provisioning latency is lower because isolation is weaker or audit continuity is broken, regulated buyers can reject the product no matter how well it does on benchmarks. The architecture must support responsiveness without removing the boundary that makes tool execution acceptable.

Cosmonic identifies concrete threats including tool poisoning through malicious MCP tools, credential theft through MCP server compromise, lateral movement from an unsandboxed agent into internal services, and manipulation of a trusted process into misusing its authority. These threats concern authority and isolation. Cosmonic also identifies CVE-2025-59528, with a CVSS score of 10.0, as a remote code execution example in its discussion of exploitable boundary failures. A working demonstration does not prove that shared-kernel isolation is enough.

An LLM acting as a trusted process can inherit database credentials, network access, and filesystem permissions. Traditional RBAC validates identity while leaving intent unchecked, so manipulation can lead the process to use authority it legitimately possesses. Sandbox isolation addresses that ambient authority at the boundary rather than assuming an authorized process will always use it appropriately.

The procurement requirements follow the workload being sold. They include SOC 2 Type II, HIPAA for healthcare workloads, GDPR for agents processing EU personal data, and the EU AI Act’s high-risk obligations for agents in HR, credit, healthcare, and critical infrastructure. A good latency result does not meet any of these requirements. Nor does persistence alone satisfy them, because its value depends on isolation, monitoring, and audit continuity being designed into the environment.

Where accountability should actually sit: making cold-start a product requirement, not an ops ticket

The organizational response is to give sandbox provisioning a product-owned SLA. The team accountable for user experience should define the requirement, while infrastructure engineering supplies the means to meet it. That makes clear how choices about throughput and cost affect the product’s promise to respond quickly and finish the work.

In practice, the instruction to “minimize cold-start” needs a concrete latency budget. The sandbox must become live within the time the interaction can tolerate. P99 should be tracked with the median, because unpredictable stalls are part of the experience, and task completion should remain part of the evaluation.

The same ownership should govern the architecture’s constraints. Isolation must follow the product’s threat model, rather than the cheapest option that passes a demo, and production agents handling untrusted code should not rely on shared-kernel containers. Statefulness should preserve an agent’s own environment and behavioral context without crossing tenant boundaries. Compliance and deployment requirements, including SOC 2, HIPAA, GDPR, and BYOC, should be confirmed against the architecture before vendor selection.

These are product decisions expressed through infrastructure. Filing them as ops work lets cost and throughput become the default measures of success, even when the user is waiting or the task remains unfinished. The misclassification is what allows cold-start latency to linger, and correcting it is what gives the problem an accountable owner.