Where Docker keeps data

The containerd image store, snapshotters and the classic graph drivers.

Where does your Docker host keep images and container filesystems, and what does each layout cost on disk? On a host installed fresh with Docker 29 the images aren't under /var/lib/docker at all. So a du -sh /var/lib/docker that reports a few hundred megabytes on a full disk proves nothing. Which directory holds them depends on which image store the daemon runs, and that depends on how the host was installed. Start by asking the daemon:

overlayfs with driver-type io.containerd.snapshotter.v1 is the containerd image store, the default for fresh Docker Engine 29 installs. Image content and container filesystems belong to containerd and live under /var/lib/containerd. A host upgraded from Docker 28 or older reports overlay2 here instead, with lines such as Backing Filesystem and Supports d_type. That's the classic graph driver. It keeps everything under /var/lib/docker and stays in place after the upgrade until someone migrates the host. You'll meet both for years, so the rest of the lesson takes them apart side by side.

A container's filesystem on the containerd store

Build the lab image and start a container from it. Its Dockerfile adds two files of random bytes (100 MB and 50 MB), so you can spot copies easily in sizes and timings:

FROM alpine:3.24
# two files of random bytes, big enough that a copy-up shows in timings and sizes
RUN head -c 100000000 /dev/urandom > /data.bin \
 && head -c 50000000 /dev/urandom > /meta.bin
CMD ["sleep", "infinity"]

Old runbooks read .GraphDriver.Data.UpperDir to find a container's files on the host. On the containerd store GraphDriver is null, and the new Storage field only names the snapshotter. The paths come from the mount itself. dockerd mounts each container's root filesystem at /var/lib/docker/rootfs/overlayfs/<id> ("Engine architecture on Docker 29" found it the same way). containerd can describe the snapshot behind it:

A snapshot is containerd's name for one filesystem layer on disk. The overlayfs snapshotter stores each one as a numbered directory, and an overlay mount stacks them. lowerdir lists the read-only snapshots, top first. 859 is Docker's init layer: placeholder files such as /etc/hosts and /etc/resolv.conf that Docker fills in per container. 858 is the image's RUN layer, and 7 is alpine's base layer, which every image built on this alpine shares. upperdir is the container's writable layer. ctr shows the same chain from containerd's side: an Active (writable) snapshot named after the container ID, whose parent is the -init snapshot. The numbers are allocation order on this VM; yours will differ.

Copy-on-write, and what the first write costs

Lower layers are never modified. The first time your container changes a file that exists in a lower layer, overlayfs copies the whole file into the upper directory and changes the copy. docker ps -s shows the writable layer's size, so you can see the cost without sudo:

Appending two bytes to /data.bin took 0.10 s the first time and nothing the second. The writable layer went from 4.1kB to 100MB: the entire file was copied up before one byte changed. Timings depend on the disk; the size jump doesn't. virtual adds the image's size. Each container pays this separately, because each has its own upper directory. So ten containers that touch the same large file store ten copies.

Since Linux 4.19, overlayfs can copy only metadata for changes such as chmod (the metacopy feature). But it's off unless the module parameter or the metacopy=on mount option turns it on, and Docker doesn't set the option. Here's the module parameter on this kernel:

N means off, and a permission change on the 50 MB meta.bin copied all of it: 150MB now. That's why the rule from "Volumes and bind mounts" in Docker for beginners matters in practice. Database files, logs, caches and anything rewritten often belong on a volume, which bypasses the overlay mount entirely.

Deletes are whiteouts

docker diff reports the change set: A added, C changed, D deleted. On disk, the upper directory holds the two copied-up files and the new note.txt. It also holds etc/issue as a character device with device number 0, 0. That's an overlayfs whiteout. It tells the merged view to hide /etc/issue from the lower layer, whose bytes are untouched. The same mechanism in an image build is why deleting a file in a later RUN doesn't shrink the image; "Slimming images and layer hygiene" in Advanced container security deals with that. Files in a writable layer also count as inodes, not only bytes. A filesystem can run out of inodes with gigabytes free, so watch both on the filesystem that holds /var/lib/containerd:

Here both are far from full. Some hosts run many images with tens of thousands of small files (language runtimes, node_modules). On those, IUse% is the number you alert on. Clean up the main VM part:

Inside /var/lib/containerd

The rest of the lesson runs on secopslog-docker-sec, where your login isn't in the docker group, so every command uses sudo. Fetch the lesson files into ~/lab/storagedrivers on that VM too (the lesson files box shows the command) and work there. A recreated sec VM may not have alpine yet, so pull it first. Then look at where containerd keeps its data and how that's configured:

containerd config dump prints the effective configuration. root holds persistent data; state (under /run) holds sockets and runtime state that disappear at reboot. The file Docker's packages install, /etc/containerd/config.toml, has both lines commented out, so the defaults apply. It also has no version line. That's why containerd 2.x warns that it migrated the config from version 0 in memory. The warning is harmless, and containerd config migrate prints an upgraded file if you want to silence it.

The content store is content-addressed: every blob is a file named after its own SHA-256 digest. The image ID that Docker 29 prints for alpine:3.24 is such a digest. The file with that name is the image index, a small JSON document listing one manifest per platform ("Image anatomy: index, manifest, config and layers" reads it in detail). unknown entries are attestation manifests, not a platform.

Two copies of every layer

The containerd store keeps each layer twice. One copy sits compressed in the content store, as pulled or built. The other is unpacked in a snapshot that containers can use. Measure what one 50 MB layer of random bytes (which doesn't compress) costs:

FROM alpine:3.24
# 50 MB of random bytes: compression cannot shrink it, so disk growth is easy to read
RUN head -c 50000000 /dev/urandom > /blob.bin

The content store grew by 48 MB: the layer as a compressed blob, which for random bytes is as large as the data. The snapshot directory grew by 104 MB, about twice the layer, because the build left more than one snapshot of it. docker buildx du reports the build-cache record for the RUN step at 100MB and counts everything under Shared. That means the cache and the image use the same snapshots, so docker builder prune can't free them while lab-blob:1 exists. That's about 150 MB of disk for a 50 MB layer. An image you pull rather than build costs the compressed blob plus one unpacked snapshot. Docker's documentation gives that double storage as the reason the containerd store needs more disk than the legacy drivers. Size your disks for it, and expect different absolute numbers on your VM, which holds other images.

The legacy graph driver, side by side

Switch this VM to the legacy store with the procedure from "Configuring the daemon safely". Back up the current daemon.json (an empty {} stands in when there's none, as on this VM). Merge the new key into the backup with jq, validate, restart, check. Merging instead of overwriting keeps whatever the host already had in the file, such as the MTU settings the lab kit writes on VPN machines. Images don't move between the stores, so save one first to load on the other side:

{
  "features": { "containerd-snapshotter": false }
}

docker info now reports overlay2 and the classic graph-driver details. The image list was empty until docker load put alpine into the legacy store. Build the same 50 MB image here:

/var/lib/docker grew from 10 to 58 MB, 48 MB for the same layer, and containerd's directories didn't move. The legacy store keeps one unpacked copy and no compressed blob. That's also why it reported CONTENT SIZE 0B in "Inspecting, exporting and cleaning up images". Now look at a container on this store:

On the legacy store .GraphDriver.Data names every directory: LowerDir (here the -init layer and alpine), UpperDir, MergedDir and WorkDir, all under /var/lib/docker/overlay2/<id>/. The whiteout looks exactly as before, because the kernel's overlayfs does the work on both stores. Each layer directory holds diff (the layer's files), link (a short name), lower (the short names of its parents), and for containers merged and work. The short names under l/ are symlinks. They keep the overlay mount options under the kernel's page-size limit when an image has many layers. Remove what you created on this store before you leave it:

Switching back, and the data left behind

Restoring the backup, which has no features key, doesn't bring the containerd store back once legacy data exists. The daemon finds /var/lib/docker/overlay2 and treats the host as an upgraded one ("Configuring the daemon safely" and "Inspecting, exporting and cleaning up images" both show that trap). Ask for the containerd store explicitly:

{
  "features": { "containerd-snapshotter": true }
}

The containerd store's images are visible again, including the lab-blob:1 built earlier, and /var/lib/docker/overlay2 is still there. Switching stores hides the other store's data; it never deletes it. On a real host upgraded from Docker 28, this is the disk-usage trap of the migration. After "containerd-snapshotter": true, every old image and container still occupies /var/lib/docker/overlay2 and /var/lib/docker/image/overlay2, invisible to docker system df. Migrate in this order. List what the host needs, switch, then pull or rebuild those images (or docker save them before and docker load after). Recreate the containers, and only then remove the old directories with Docker stopped. Docker also has an experimental automatic migration ("containerd-migration": true in features). It switches only when no containers exist and the images are below a size threshold, and the docs recommend starting fresh instead. This lab VM never held anything important in the legacy store, so it removes the directories at once and restores the backup:

One daemon option rules the containerd store out entirely: userns-remap disables it. A host that needs user-namespace remapping runs the legacy store ("User-namespace remapping" in Advanced container security covers the trade-off). Rootless Docker keeps its own data under the user's home directory; "Rootless Docker" in Advanced container security covers it.

Moving containerd's root

data-root in daemon.json moves only dockerd's directory, as "Configuring the daemon safely" showed. To put images and container filesystems on another disk, you move containerd's root as well. Stop or remove every container first, and check that docker ps -q prints nothing. With live-restore on, stopping docker.service leaves containers running on their shims, and the copy would race their writes to the upper directories. Then you stop both daemons. The copy must keep hard links, ACLs and extended attributes, which overlay layers use (rsync -aHAX). Plan it as a maintenance window. In production /srv/containerd would be the mount point of the new disk:

After the restart the images are still listed, and the new container's upperdir is under /srv/containerd. Until you delete it, the old /var/lib/containerd is your rollback. It has one cost: switching back loses every image pulled and every container change made since the move. Once the new location has run for a while, remove it to free the space. On a real disk, make sure containerd never starts before the disk is mounted. Otherwise it creates a fresh, empty root on / and fills the root filesystem. A systemd drop-in prevents that:

[Unit]
RequiresMountsFor=/srv/containerd

Each daemon needs its own root: never point two containerd instances (or two hosts over NFS) at the same directory. The lab moves back:

d_type and XFS

Overlayfs needs the backing filesystem to report each directory entry's file type (d_type); it uses it to recognise whiteouts. ext4 always supports it. XFS supports it only when formatted with ftype=1, which mkfs.xfs has defaulted to for years. But old RHEL and CentOS 7 era hosts were often formatted with ftype=0. The classic overlay2 check is what prints Supports d_type: true in docker info. To see what happens without it, the lab formats an XFS image with ftype=0 and points each store at it. (It needs the deprecated V4 format to allow that.)

Nothing failed, and that's the problem. With ftype=0 under data-root, dockerd didn't select overlay2. It tried fuse-overlayfs (not installed, the one error line). Then it settled on vfs, which logs at info level only. vfs has no copy-on-write. Every layer and every container is a full copy of everything below it. So disk use multiplies and container starts slow down. On this store the daemon gives up on overlay2 without a d_type message in its log. The place you catch it is docker info after provisioning: alert on any driver other than the one you expect. Docker's deprecation notes add the other half. Since Engine 23, a daemon told explicitly to use overlay2 (storage-driver in daemon.json) on such a filesystem refuses to start; automatic selection, as here, skips it. Stop the daemon and restore the backup before the next test, so dockerd starts on its defaults again:

Now try the containerd store on the same filesystem. Point containerd's root at the XFS mount, restart containerd, and start dockerd:

containerd itself starts, but its overlayfs snapshotter plugin is in state error, and the kernel log says why: upper fs needs to support d_type. dockerd then refuses to start. Its journal names the cause in a warning (does not support d_type ... reformat with ftype=1). The line that ends the start is an unrelated-looking buildkit error about a missing leases service, and that's what you'd see first in systemctl status. So the containerd store fails loudly where the legacy store degrades quietly. The fix is the same for both. Check xfs_info <mountpoint> | grep ftype before you put Docker or containerd data on XFS. If it says 0, reformat with ftype=1 (the default in current mkfs.xfs). The restore step puts containerd's original config back, removes the test filesystem and starts both daemons.

Clean up the lab directory on the sec VM:

Try this

On the main lab VM, rebuild the lab image and start two containers from it under different names. Append to /data.bin in only one of them, then run docker ps -s again. Predict both writable-layer sizes first, using the 4.1kB and 100MB values above, then check whether the untouched container paid anything.

Bottom line

Keep write-heavy files on volumes, and size your disks for the store you actually run. Two appended bytes grew a writable layer from 4.1kB to 100MB, and one 50 MB layer cost about 150 MB on the containerd store. The quiet failure on real hosts is the old XFS volume with ftype=0, where the legacy store fell back to vfs with only an info-level log line. Read the driver from docker info after every provisioning run; don't trust the install docs for it.

Related