Copy-on-Write Memory Forking for Sandbox Cloning

There is a version of this problem that sounds completely made up.

Someone visits a website, clicks "fork," and within two seconds they have their own copy of a running 8-gigabyte virtual machine. Not a blank slate they have to configure. Not a loading spinner while memory gets duplicated. An exact, live clone. The Python interpreter is already loaded. The dependencies are already imported. The JIT is already warm.

How is that not magic?

It is not magic. It is a very old trick the Linux kernel has been doing since before most of us were writing production code. And once you see how the pieces fit together, the two-second number stops feeling impressive and starts feeling almost obvious.

Almost.

How the kernel implements COW at the page level

Start with the simplest version of the problem.

You have a process. It has 4 gigabytes of memory. You want a copy of it. The naive approach: allocate 4 gigabytes of new RAM, copy every byte, hand the copy to a new process. That takes time proportional to how much data you have. Double the memory, double the wait. At some point, the math just does not work anymore.

Copy-on-write (COW) refuses to play that game.

When the Linux kernel forks a process, it duplicates the page table, not the underlying physical pages. Think of the page table as an index. It maps "this address in your process" to "this physical location in RAM." Duplicating the index is cheap. The actual data it points to? Still sitting in the same physical RAM, untouched, shared between parent and child.

Both processes now point to the same physical memory. The kernel marks those shared pages read-only and keeps a reference count. As long as nobody writes, nothing moves.

The moment a process tries to write to one of those pages, the CPU traps into the kernel. This is the COW page fault. The kernel:

• Allocates a new physical page
• Copies 4 kilobytes of data from the shared page
• Updates the faulting process's page table to point at the new, private copy
• Decrements the reference count on the original page

If the reference count drops to one (meaning only one process still shares that page), the kernel skips the allocation and just marks the page writable in place. No copy needed.

The whole trap-and-copy sequence takes microseconds. User code resumes as if nothing happened.

That raises an important question: what does this actually change about clone time? Clone time is now proportional to the number of mappings (metadata), not the number of bytes. A 2-gigabyte guest and a 4-gigabyte guest clone in roughly the same time. The actual byte-copying is deferred to the precise moment a page is written, one 4-kilobyte page at a time.

One wrinkle worth knowing: hugepages change the arithmetic. Instead of 4-kilobyte pages, hugepages give you 2-megabyte pages. One fault covers 512 times as much memory, which cuts the total fault count dramatically for memory-heavy workloads. But this requires the snapshot to be built with hugepage backing. It is a construction-time decision, not a free upgrade you toggle later.

Where COW forking creates real overhead: TLB shootdowns

Here is the part people often skip over, and they shouldn't.

Every CPU core has a Translation Lookaside Buffer (TLB). It is a cache of recent page table lookups. When the kernel modifies a page table entry during a COW fault, any other core that cached the old mapping now has stale data. The kernel has to tell those cores to flush their caches. This is a TLB shootdown, and it involves broadcasting interrupts to other CPU cores.

The cost of a shootdown is proportional to the number of cores that share the affected address space. Not to the size of the page. One 4-kilobyte write can trigger an interrupt broadcast across dozens of cores.

For forked processes with separate address spaces (which is the sandbox case), the cross-process shootdown problem is less severe. The parent's mapping to the original page stays valid. The child is the one diverging. But in multithreaded guests where many CPUs share the same address space, high write concurrency means lots of faults, lots of shootdowns, and latency that compounds in ways the "microseconds per fault" framing obscures.

Selective TLB shootdown, where the kernel consults which cores actually have the faulting page cached and sends interrupts only to those cores, measurably improves throughput for write-intensive workloads. It is not a silver bullet, but it is the right direction.

The practical design implication: isolating each sandbox VM in its own address space, as Firecracker does, sidesteps the worst-case shootdown scenario. Each microVM is a separate process. Separate address spaces. COW faults in one do not shower interrupts across the address space of another.

Why VM snapshots are the right unit for sandbox cloning

Why not just fork a process? Or spin up a container?

A process fork shares the host kernel and libc. A container shares the host kernel. Neither gives you hardware-level isolation per clone. For untrusted code execution, that matters.

A Firecracker microVM snapshot captures the full guest state: memory, CPU registers, device state. Everything the guest was in the middle of doing. When you restore that snapshot, you get a VM that resumes from exactly where it was frozen. Python already loaded. Libraries already imported.

The restoration mechanism is where COW enters: restoring a snapshot using mmap(MAP_PRIVATE) means reads go directly to the snapshot file pages. Zero copy until first write. Writes trigger COW allocation of new pages, diverging from the snapshot without modifying it.

The snapshot is immutable. Reusable. Many independent clones can all be backed by the same on-disk image simultaneously. Each clone diverges privately as it runs, but the shared baseline costs almost nothing per clone.

It is worth considering what this means for cold-start time. A restored microVM does not boot from scratch. It resumes from a running state. The expensive work (loading the interpreter, importing packages, warming the JIT) was done once, when the snapshot was created. Every clone inherits that work for free.

userfaultfd: extending COW semantics across process and host boundaries

Normally the kernel resolves page faults itself. It finds the page, maps it in, resumes the faulting thread. The process never knows anything happened.

userfaultfd (UFFD) flips this. Instead of handling the fault itself, the kernel hands it to a user-space handler process, waits, and resumes the faulting thread once the handler installs the page. User space is now in charge of deciding what memory looks like and when it loads.

Linux introduced UFFD in 2017. Before that, there was no official user-space API to intercept and control memory loading.

Why does this matter for sandboxes? In a VM-fork scenario, UFFD lets a child VM read its memory directly from a live parent VM, on demand. A UFFD handler registers itself to manage the child's memory. When the child faults on an uninitialized page, the handler copies that page from the parent and installs it. Pages the child never reads are never transferred.

The child inherits the parent's full address space without paying the cost of copying it upfront. The pages that never get touched never get copied. At all.

UFFD and kernel COW are complementary, not alternatives. UFFD is the streaming source that lazily populates a child's address space from a remote or on-disk parent. Kernel COW is what makes local sharing and per-page divergence cheap once pages are installed. A page streamed in by UFFD then diverges under the same kernel COW rules as any other shared page.

Firecracker gives operators a choice between OS-managed page faults and a UFFD backend. Firecracker added support for /dev/userfaultfd on Linux kernels 6.1 and later. Pre-allocating the in-kernel file descriptor table yields a measurable reduction in snapshot restore times for medium to large microVMs: somewhere in the 30 to 70 millisecond range, according to the brief's figures. At the scale of a two-second budget, that is not nothing.

How CodeSandbox clones 4–12 GiB VMs in under 2 seconds

Here is where the theory meets a real product requirement with a real deadline.

CodeSandbox's constraint: any visitor can fork a running development environment and have an exact copy within two seconds. The guest VMs carry 4 to 12 gigabytes of memory. Memory can be copied at roughly 2 to 3 gigabytes per second. Do the math. For larger guests, naive copying does not fit in the budget. Not even close.

Their approach combines two techniques.

First: eager incremental serialization. Memory changes are written to disk continuously, so most of the snapshot is already on disk when a fork is requested. The fork does not have to wait for a full serialization step. Most of that work is already done.

Second: UFFD-based child-reads-from-parent. A uffd-handler process registers to manage the memory of the parent VM and all its children. When a child VM faults on a page, the handler copies it from the parent's live memory and installs it in the child.

The implementation detail here is clever. For each VM, uffd-handler creates a memory-backed file via memfd_create, sends the file descriptor to Firecracker, and Firecracker mmaps it as the guest's memory. The guest never knows its pages are being served on demand. From inside the VM, everything looks normal.

Pages the child never touches are never copied. The child inherits the full address space of the parent at the cost of the UFFD registration and metadata, not the cost of the data.

The two-second wall is met not by copying faster but by copying less. That reframe is the whole insight.

What production sandbox tools have built on top of this model

Four independent projects landed on the same architecture. That kind of convergence is worth paying attention to.

forkd (deeplethe, 2026) boots a parent Firecracker VM once with a warmed runtime (Python, dependencies, a loaded ML model), pauses it to disk, then spawns children by mmapping the parent's memory image with MAP_PRIVATE. Each child is a separate Firecracker process with KVM isolation. It benchmarked at 100 children in roughly 100 milliseconds from a warm parent. The measured COW overhead at that fan-out is 0.12 MiB per child on top of a 512-MiB warmed Python-plus-numpy parent image. At scale, the binding constraint is not memory. It is vCPU count and process limits.

E2B / ZeroBoot identifies the bottleneck as memory copying during snapshot restore and eliminates it with MAP_PRIVATE. Reads go directly to snapshot pages. Writes trigger COW allocation. The Python interpreter and standard libraries shared across all sandboxes are never duplicated until actually written. Startup requires almost no memory copying at all.

PandaStack (2026) restores every sandbox from a baked snapshot and achieves a p50 around 179 milliseconds for a full create. Same-host forks come in well under a second, regardless of guest RAM size.

Sub-millisecond HN POC (March 2026): boots Firecracker once with Python and numpy loaded, snapshots the full VM state, then backs every subsequent execution with a MAP_PRIVATE mapping of that snapshot. Each sandbox starts from an already-running Python process inside a real VM, runs its workload, and exits.

The common pattern across all four: the snapshot is immutable and reused. COW is what lets N simultaneous clones share it without interfering with each other or with the snapshot.

The limits COW forking does not eliminate

It would be easy to read everything above and conclude that COW forking is a free lunch. It is not.

COW defers cost. It does not eliminate it. Every written page eventually pays the fault-and-copy penalty. A write-heavy, long-lived clone accumulates COW faults across most of its address space. Eventually, it carries roughly the same memory footprint as a full copy. The sharing advantage is largest when:

• Clones are short-lived (total fault cost stays below the upfront copy cost)
• Workloads are read-dominated or touch only a small fraction of inherited pages
• Many clones share the same snapshot simultaneously (per-clone overhead stays near 0.12 MiB rather than growing to the full guest size)

TLB shootdown overhead in multithreaded guests can erode the fault-handling speed advantage. Workloads with high write concurrency should measure actual fault latency rather than assuming microsecond traps across the board.

Hugepages help amortize fault count but require the snapshot to be built with hugepage backing. You can't retrofit this decision.

Diff-snapshot chains, as in forkd v0.5, extend COW semantics across layers of specialization. A base snapshot, then a Python-layer snapshot on top, then a numpy-layer snapshot on top of that. The memory sharing is real and the economics are good. But the daemon must walk the chain at spawn time, and chain depth adds latency. At some chain depth, the savings from sharing common layers stop justifying the overhead. Where that crossover sits depends entirely on your workload.

The honest framing: the cost model here is clear enough to instrument. Measure your write patterns before assuming sharing holds. The technique is powerful, but it is not a substitute for understanding what your workload actually does.

That two-second clone time is real. The physics behind it is real. But so are the edge cases where the physics work against you.