Layers and the build cache

Why order matters, what busts the cache, what image sizes mean and where layers live.

The order of the instructions in a Dockerfile decides how much of each rebuild BuildKit can skip. Get it wrong and a pipeline reinstalls every dependency on every commit, even for a one-line README fix. Get it right and a code change rebuilds one small layer. Here you'll measure both. Two ideas explain the result. An image is a stack of layers. And BuildKit reuses a cached layer only while everything that went into it is unchanged. You'll also see what the image sizes mean, and why deleting a file in a later step doesn't remove it.

Use the main lab VM and unpack the lesson files into ~/lab/layers. They contain the small Express service from "Writing a Dockerfile" (renamed payments-api) with its committed package-lock.json and a .dockerignore. There are three Dockerfiles: one in the right order, Dockerfile.bad in the wrong order, and a third for the last section.

FROM node:26-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
USER node
EXPOSE 3000
CMD ["node", "server.js"]

An image is a stack of layers

Each Dockerfile instruction that changes files produces a layer. Those are COPY, ADD, RUN, and WORKDIR when it creates the directory. A layer is a set of file additions, changes and deletions relative to the layers below it. Instructions such as ENV, USER, EXPOSE and CMD only change the image configuration and add no files. FROM adds no layer of its own. It brings in the base image's layers. Layers are identified by the digest of their content, so two images built FROM node:26-alpine store the base layers once and share them. When a container starts, Docker adds a thin writable layer on top, private to that container, as described in "Images, containers and the core commands".

Build once and list the layers

The first build uses --no-cache so every step runs, as it would if you built on a fresh machine. As in "Writing a Dockerfile", the plain progress format is shown and npm's update notice is cut.

npm also reports 1 high severity vulnerability. The committed package-lock.json pins 68 packages, and an advisory against one of them came out after the lockfile was written. That's the other side of a lockfile. It keeps every build identical, including the vulnerable version, until you regenerate it on purpose. Run npm audit to see which package it is.

docker history shows you the layers of an image, newest first, with the instruction that created each and the size it added.

The top seven rows are this Dockerfile. Everything from CMD ["node"] down came from the base image, which was built four days before this run. Your sizes and ages will differ. The configuration-only instructions (CMD, EXPOSE, USER, ENV) show 0B. RUN npm ci added 9.95MB, the largest layer of your own. The bottom row shows the Alpine root filesystem for aarch64, because the lab VM is an arm64 machine. On an amd64 host the same tag resolves to the amd64 variant, and that row names an x86_64 archive. <missing> in the IMAGE column only means those intermediate layers have no image ID of their own. It isn't an error. The COMMENT column shows that BuildKit built every layer, including the base image's.

How the cache decides

On each build, BuildKit walks the instructions in order. For each one it asks whether it already has a result for this exact step on top of this exact parent. For RUN, the key is the command text. BuildKit doesn't look at what the command would download. For COPY and ADD, the key includes a checksum of the files being copied. The first instruction whose key changed is executed again. So is every instruction after it, because each one builds on the result of the one before.

Append a comment to server.js and build again. The comment carries the current time, so the change is new every time you try this.

WORKDIR, the manifest copy and npm ci come back CACHED because you changed none of their inputs. Only COPY . . ran, since server.js is part of what it copies. The plain output lists steps in the order BuildKit finished resolving them, which isn't always file order. The [n/5] numbers give the Dockerfile order.

The wrong order

Dockerfile.bad copies everything first and installs afterwards.

FROM node:26-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
USER node
EXPOSE 3000
CMD ["node", "server.js"]

Build it once, then change server.js again and rebuild, as you did above.

This time COPY . . sits below the install, so the edited file changes its key. Then npm ci runs again for a change that has nothing to do with dependencies. Here that costs about two seconds, because the app has a single dependency. A real service with a few hundred packages pays minutes per build, on every commit. The rule follows directly. Copy what changes rarely (dependency manifests, then the install) before what changes often (the code). You'll find the same pattern in every ecosystem: requirements.txt before pip install, go.mod and go.sum before go mod download, pom.xml before the Maven dependency step.

A cached step is reused however old its result is. A RUN apt-get update && apt-get install -y curl step keeps its cached result until something below it changes, however old the package lists get. Pin versions where you need repeatability. When you deliberately want fresh results, rebuild with --no-cache, or --pull to also check for a newer base image. A scheduled rebuild that picks up security fixes is the usual example.

What the sizes mean

All three images report 270MB of DISK USAGE, but they don't use 810MB between them. DISK USAGE counts every layer an image needs, including the shared base, and the images here differ only in their top layers. CONTENT SIZE (70MB) is the compressed size of the layers, roughly what a pull or push transfers. docker system df totals what's actually stored, counting shared layers once. The image count and the totals include everything on the VM, so yours will differ.

The Build Cache row is BuildKit's store of step results, including results no current image uses. It's kept until BuildKit's garbage collection trims it or you remove it with docker builder prune. That command clears cache for every project on the host, so on a shared machine you should leave it to whoever looks after the host.

Where the layers live

On a fresh Docker 29 install, image content is stored by containerd, not by Docker's older graph drivers. You can see it in three places. The storage driver reports overlayfs with driver-type io.containerd.snapshotter.v1. Layer content sits under /var/lib/containerd. And /var/lib/docker has no overlay2 directory. Hosts upgraded from older versions keep the classic overlay2 driver until they migrate. "Where Docker keeps data" in Docker in depth explains both layouts and how to tell which one a host uses.

A deleted file is still in the image

Layers only add. A later instruction that deletes a file records the deletion in its own layer and hides the file from the final filesystem. But the earlier layer still contains it, and anyone with the image can extract it. The third Dockerfile copies a stand-in key file, uses it, and deletes it.

FROM alpine:3.24
COPY fake-key.txt /tmp/fake-key.txt
RUN wc -c /tmp/fake-key.txt
RUN rm /tmp/fake-key.txt

The COPY layer still holds the 3 KB file (12.3kB on disk, rounded up to filesystem blocks). The rm layer didn't shrink anything. It added a few kilobytes of its own. Keep your secrets out of build instructions entirely. "Build-time secrets and how images leak them" in Advanced container security shows how to extract such a file and the build features that avoid the problem.

Clean up

Your turn

Run touch server.js, which changes the file's timestamp but not its content, and rebuild with docker build -t lab-layers:2 .. The COPY key is a checksum of the copied files, so predict whether COPY . . comes back CACHED before you look. Then append one more comment with the same echo and compare the two builds.

Takeaway

Put the dependency manifests and the install above COPY . ., and a code edit rebuilds one small layer instead of rerunning npm ci. Two things catch teams later. A cached RUN step is reused however old it is, so security fixes need a deliberate --no-cache or --pull rebuild. And a file removed by a later rm still sits in its COPY layer, as the 12.3kB row in lab-layers:secret showed.

Related