Copy-on-write branching for AI training data
· Amulet Labs
Copy-on-write branching creates a new writable view of existing data without copying it: both branches share the same underlying bytes, and only changed data gets its own storage when one side writes. For AI training and evaluation — where every experiment wants its own modifiable copy of a large shared dataset — it turns the most expensive operation in the pipeline into a millisecond one.
How does copy-on-write actually work?
A workspace is a tree of references to immutable data blocks. Forking the workspace copies the references, not the blocks — an operation whose cost is independent of data size. Reads on either side resolve to the same shared blocks; when a branch writes, just the affected data is written newly for that branch while everything else stays shared. On Amulet a fork returns in about 3 ms with zero bytes copied, whether the workspace holds 4 GB or 40 TB.
Why does AI training need branching?
Because experiments are branches, whether your storage models them or not. A typical week of training work wants to: freeze a known-good dataset before a LoRA run, give each hyperparameter sweep member an isolated writable copy, let an eval harness mutate fixtures without contaminating the source, and roll everything back when a preprocessing bug is found mid-run. With plain storage each of those is either a full copy (slow, doubles cost) or shared mutable state (fast, and eventually someone corrupts the dataset for everyone). Copy-on-write branches make isolation the cheap default rather than an expensive discipline.
What does a thousand forks cost?
Approximately one dataset. Forks share every unchanged byte with their base, and identical bytes are deduplicated within an account, so a thousand eval environments forked from one terabyte store roughly one terabyte plus whatever each environment writes. On Amulet this interacts with pricing directly: billing follows actively used data, and a fork adds nothing to the bill until it diverges.
What about rollback and reproducibility?
Copy-on-write pairs naturally with immutable history. Amulet captures workspace state as content-addressed commits, so "the dataset exactly as it was before Tuesday's run" is a revision you restore — byte for byte — rather than a backup you hope exists. Every commit records which actor, session, and run produced it, which makes experiment provenance a lookup: given a model artifact, walk back to the precise data state that trained it.
How is this different from EBS or ZFS snapshots?
Block-device snapshots (EBS, ZFS, LVM) are infrastructure primitives: per-volume, attached to one machine, restored by an operator. CoW branching at the workspace level is a product primitive: any job or agent can fork by API in milliseconds, mount its branch from anywhere in under 100 ms, and merge or discard results deliberately. Comparing against network filesystems is starker still — EFS has no branching at all, so every isolated environment means copying bytes at ~$0.30/GiB-month.
Frequently asked questions
What is copy-on-write branching?
Copy-on-write (CoW) branching creates a new writable view of existing data without copying it. Both branches share the same underlying bytes; only when one side writes does the changed data get its own storage. A fork is therefore near-instant and initially free, no matter how large the dataset.
Why does AI training need copy-on-write branches?
Training and evaluation constantly need isolated variations of shared state: freeze a dataset before a run, give each experiment or eval worker its own writable copy, and roll back cleanly when something breaks. Copying terabytes per experiment is slow and expensive; CoW branches make each variation a millisecond operation that costs nothing until it diverges.
Does a copy-on-write fork cost extra storage?
Not until it changes. A fork shares every unchanged byte with its base, so a thousand forks of a terabyte dataset still store roughly one terabyte plus whatever each fork writes. On Amulet, that also means forks are free at creation — billing follows actively used data, and unchanged bytes are shared and deduplicated.