File Systems Considerations for a microVM Sandbox

When you build an agent sandbox, one of the next things to consider is to give it a file system, so that the agent can read and write files on it. There is also a need for snapshotting, so that a sandbox can be recovered after a crash or rolled back to an earlier state. The file system underneath the sandbox largely decides how good that snapshotting can be. The two popular choices I have seen in real-world usage are Btrfs and OverlayFS. Perplexity, for example, seems to have chosen Btrfs for their sandboxes, as described in their post on secure and efficient runtimes for long-running agents. Before comparing them, it helps to look at what Linux gives us by default and why it falls short.

What Linux has by default

ext4 is the default file system on most Linux distributions. It stores file data in blocks and tracks them through inodes, which live in inode tables on disk. Modifying a file means overwriting its blocks in place. If the system crashes halfway through, you can end up with a half-overwritten file. ext4 has a journal, but by default it protects the file system’s metadata, not the contents of your data.

ext4 also has no built-in notion of a snapshot. The naive way to snapshot a directory is to copy it, and a copy is a physical one: every byte is read and written again to a different place on disk. That is slow, it costs as much space as the original, and it has the same crash problem. If the sandbox keeps writing while the copy runs, or the machine dies in the middle of it, the copy is not a consistent picture of any single moment in time. You can bolt snapshots on from below, for example with LVM, but then the block layer is doing the work and the file system does not know anything about it. So we want a file system that is designed around snapshotting, which is where the other file systems come in.

Btrfs and copy-on-write

Btrfs (B-tree file system) is a modern copy-on-write (CoW) file system for Linux. It combines snapshots, built-in volume management, checksums, and fault tolerance in one place. Everything else it does follows from the copy-on-write design, so that is the part worth understanding properly.

What copy-on-write means

Copy-on-write is an idea borrowed from virtual memory management. When you fork a process, the parent and the child share the same physical RAM pages, and nothing is duplicated up front. Only when one of them writes to a page does the operating system allocate a new page, copy the data over, and let the writer modify its own copy. The sharing lasts exactly as long as nobody writes.

Btrfs applies the same idea to disk. If you snapshot a 10 GB folder, the original and the snapshot point at the same physical disk blocks, and no data is duplicated. The two only diverge when one side is modified.

How a write works

File systems and storage hardware read and write fixed-size blocks, typically 4 KB, not individual bytes. Even if you change a single byte, the whole block has to be handled. What differs is where the result goes.

On a traditional file system like ext4, the system reads the 4 KB block into RAM, changes the byte there, and writes the whole block back to the same location on disk, overwriting the old one. This is an in-place update. If the power cuts out mid-write, the block is left half old and half new.

On Btrfs, the first two steps are the same, but the modified block is written to a brand-new location, and the old block is left untouched. Suppose a file contains “HELLO” in Block A, and the metadata says that the file points to Block A. You edit it to “HEAVEN”. Btrfs finds free space (Block B) and writes “HEAVEN” there. Only once that write is safely finished does it update the metadata to say the file now points to Block B. After that, Block A is marked free, unless a snapshot is still referencing it. Because the old data is never overwritten before the new data is complete, a crash at any point leaves you with either the old version or the new version, never a mixture of the two.

The pointer update is itself a copy-on-write operation. Btrfs stores its metadata in B-trees, and a block that is shared with a snapshot cannot be modified in place either. So Btrfs copies the affected metadata node, applies the change to the copy, and then copies each parent node up the tree so that it points to the new child, all the way up to the root. The last step is switching the root pointer, which is a single small write.

Why snapshotting is so much better with copy-on-write

Take a snapshot on ext4 and you have to copy the data. The time it takes and the space it uses both grow with the size of the directory, and the copy is only consistent if nothing is writing while it runs. Ten snapshots of a 10 GB workspace cost roughly 100 GB and a lot of I/O.

Take a snapshot on Btrfs and almost nothing happens. A snapshot is a new subvolume whose root points at the same tree of blocks as the original. Nothing is read or copied, so it takes about the same time whether the workspace is 10 MB or 10 GB, and it takes almost no extra space at the moment it is created. Every block is now referenced twice, and Btrfs keeps a reference count on it. Space is only consumed as the two sides diverge: when the sandbox modifies a file, only the changed blocks and the metadata path above them are written to new locations, while everything untouched stays shared. The snapshot keeps pointing at the old blocks, and the live file system points at the new ones. A block is only freed once nothing references it anymore.

Because the snapshot is taken by capturing a root pointer, it is also consistent by construction. It represents the file system exactly as it was at one committed moment. For a sandbox, this means we can snapshot frequently, restore after a crash by switching back to an earlier snapshot, and branch a fresh sandbox from an existing one cheaply, all without paying for full copies.

The trade-off

Copy-on-write is not free. If an application makes many small random writes into the middle of a large file, such as a PostgreSQL database or a virtual machine disk image, every write lands in a new location, and the file’s blocks end up scattered around the disk instead of laid out sequentially. That fragmentation hurts performance. On everyday workloads, ext4 is also generally faster and lower latency than Btrfs for plain reads and writes.

Btrfs lets you turn off copy-on-write for a specific file or directory with chattr +C, which is useful for performance-critical paths like a database folder. The price is that those files lose the benefits that depend on CoW, such as checksums and cheap snapshots of their data. For a sandbox whose main job is to be snapshotted and recovered, that is a good reason to leave CoW on for the workspace and to be deliberate about which directories, if any, you exempt.

OverlayFS

OverlayFS takes a completely different approach from Btrfs. It is not a disk file system with its own block layout and its own copy-on-write engine. It is a union mount: a thin layer that sits on top of two existing directories, each of which can live on any ordinary file system such as ext4, and presents them to the sandbox as one merged directory.

The two directories are called the lower directory and the upper directory. The lower directory holds the base image, for example a clean root file system or a prepared workspace, and is mounted read-only. The upper directory starts out empty and is where all writes go. OverlayFS merges the two into a single view: reading a file that only exists in the lower directory serves it straight from there, and reading a file that has been created or changed shows the version in the upper directory instead. Deleting a file that exists in the lower directory does not touch the lower directory at all, since it is read-only; OverlayFS instead writes a small marker called a whiteout into the upper directory, which tells the merged view to hide that file.

The mechanism for a write is called copy-up. The first time the sandbox modifies a file that only exists in the lower directory, OverlayFS copies the entire file up into the upper directory, and from then on all reads and writes to that file are served from the upper copy while the lower one is left untouched. This is where OverlayFS differs from Btrfs in an important way: the granularity is a whole file, not a 4 KB block. Copying up a small config file costs almost nothing, but copying up a multi-gigabyte file the first time you touch even one byte of it is expensive, and that cost is paid on every fresh sandbox that touches that file for the first time.

Snapshotting with OverlayFS is really just snapshotting the upper directory, since the lower directory never changes. That upper directory tends to be small, because it only holds the files the sandbox actually modified, so backing it up or diffing it against a clean state is cheap. Starting a new sandbox from the same base image is also cheap: mount the same read-only lower directory again with a fresh, empty upper directory, and you have an independent sandbox in roughly constant time, without needing any copy-on-write support from the underlying disk file system at all. This is likely why OverlayFS is attractive for a sandbox fleet that spins up many short-lived, mostly-identical sandboxes from the same base image. Arrakis uses OverlayFS for their sandbox file system for this kind of setup.

The trade-off is the copy-up cost on first write, and the fact that the crash-safety of the whole stack depends on whatever file system the upper directory actually sits on. OverlayFS itself has no journal and no checksums; it inherits the guarantees, and the weaknesses, of the file system underneath it.