BuildKit builds: stages, cache and mounts

Multi-stage builds, parallel stages, cache, bind, secret and SSH mounts.

A Go service ships as a 493MB image. The binary inside it is 5MB. The rest is the Go toolchain, Alpine's package manager and the source tree. They're all copied in because the Dockerfile compiles and runs in the same image. Every pull you make moves those bytes, and every scanner you run reports their CVEs. Every code change recompiles the whole standard library, because one COPY . . invalidated the cache. You'll fix all three with BuildKit, the builder behind docker build. The tools are multi-stage builds, stages that run in parallel, and RUN --mount for caches, sources, secrets and SSH agents.

Use the main lab VM, secopslog-docker. The Go module, the Dockerfiles and .dockerignore are in the lesson files download; unpack them into ~/lab/multistage. No host settings change.

BuildKit is already the builder

Docker Engine 29 builds with BuildKit by default. docker build is handled by the Buildx plugin, which sends the build to a builder. On a fresh install that's the default builder, BuildKit running inside dockerd:

docker in the DRIVER column means the builder lives in the daemon and writes straight into the local image store. BuildKit v0.34.0 is the version embedded in Engine 29.9.2. "Buildx builders, outputs, cache and Bake" adds builders that run in their own containers. Everything here uses the default one. The legacy builder still exists behind DOCKER_BUILDKIT=0. It's deprecated, and it doesn't understand any of the RUN --mount options below.

Stages and targets

Each FROM starts a stage. A stage can start from an image or from an earlier stage, and COPY --from=<stage> copies files out of another stage. Only the stage you build (by default the last one) becomes the image. The Dockerfile you'll use has five:

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS base
WORKDIR /src
ENV CGO_ENABLED=0
 
FROM base AS build
RUN --mount=type=bind,target=. \
    --mount=type=cache,id=lab-go-build,target=/root/.cache/go-build \
    go build -trimpath -ldflags="-s -w" -o /out/lab-app .
 
FROM base AS test
RUN --mount=type=bind,target=. \
    --mount=type=cache,id=lab-go-build,target=/root/.cache/go-build \
    go test ./... && touch /tmp/test.ok
 
FROM base AS vet
RUN --mount=type=bind,target=. \
    --mount=type=cache,id=lab-go-build,target=/root/.cache/go-build \
    go vet ./... && touch /tmp/vet.ok
 
FROM scratch AS checks
COPY --from=test /tmp/test.ok /
COPY --from=vet /tmp/vet.ok /
 
FROM scratch AS runtime
COPY --from=build /out/lab-app /lab-app
USER 65532:65532
ENTRYPOINT ["/lab-app"]

base holds what you need for every Go step. build, test and vet each start from base and don't depend on each other. checks collects marker files from test and vet. runtime starts from scratch, an empty filesystem, and receives one file from build. The golang image is pinned by tag and digest as in "Tags, digests and promotion". The final stage runs as the numeric user 65532, because scratch has no /etc/passwd to look a name up in ("Run as non-root" in Advanced container security covers numeric IDs). The next sections explain the RUN --mount lines. Build the default target:

Read the step names you got. (The cut lines include this VM's first pull of golang:1.27-alpine.) BuildKit ran [base 1/2], [base 2/2], [build 1/1] and [runtime 1/1], and nothing from test, vet or checks. It builds the graph of stages the target needs and skips the rest. So a test stage in the same Dockerfile costs you nothing in a plain image build. [build 1/1] took 6.5s, almost all of it compiling the standard library packages that net/http pulls in. Your times depend on the CPU; the order of magnitude is what matters here. For comparison, here's the single-stage version that most projects start with:

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c
WORKDIR /src
COPY . .
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /usr/local/bin/lab-app .
ENTRYPOINT ["lab-app"]

Same source, same binary: 7.74MB against 493MB on disk. (DISK USAGE counts the unpacked layers, CONTENT SIZE the compressed ones; "Inspecting, exporting and cleaning up images" explains both columns.) The builder stages still exist in the build cache on this machine. But nothing in lab-app:1 refers to them, so they're never pushed. scratch works here because the binary is static (CGO_ENABLED=0). Choosing between scratch, distroless and Alpine is the subject of "Minimal bases: distroless, scratch, static and Alpine" in Advanced container security.

Cache mounts versus the layer cache

The layer cache ("Layers and the build cache" in Docker for beginners) is all or nothing per step. If a step's inputs changed, the step runs again from scratch. In Dockerfile.single, any edit changes COPY . ., so go build starts with an empty Go build cache every time. Make a one-line change and rebuild both versions, so you can time them:

That's 11.8 seconds for the single-stage rebuild and 1.2 seconds for the multi-stage one. The difference is --mount=type=cache,id=lab-go-build,target=/root/.cache/go-build on the RUN line. A cache mount is a directory BuildKit keeps between builds and mounts into the step while it runs. The step still re-ran (its source changed). But Go found the compiled standard library from the previous build in its own cache, and only recompiled main. The mount's contents never become part of a layer, so the image doesn't grow. id names the cache, and steps that use the same id share it. That's why the test and vet stages below reuse the compiler output. Without id, the target path is the id.

Cache mounts are build cache records of type exec.cachemount, 90.94MB for lab-go-build here. The other two records come from other lessons' builds on this VM. docker builder prune removes cache mounts like any other build cache. Treat them as an accelerator, not storage. A cold CI runner starts with an empty one, and your build must still be correct. Some package managers lock their cache (apt, for example). For those, add sharing=locked so parallel builds wait for each other instead of corrupting it; the default is sharing=shared.

The other mount on those lines, --mount=type=bind,target=., makes the build context visible at the working directory for the length of the step. It's read-only by default. Nothing is copied into a layer, so the source tree never ends up in the build stage's filesystem. The step's cache key still covers the files it can see.

Stages in parallel

test and vet both depend only on base. Build the checks target, which needs both. -o type=cacheonly tells BuildKit to run the build without exporting an image, since a stage of marker files isn't worth keeping. "Buildx builders, outputs, cache and Bake" covers output types. time measures the whole build:

Step #7 (vet) and step #8 (test) both start, and each prints ... while the other one is logging. They finish within 0.1 seconds of each other at about 16 seconds. The whole build took 16.4 seconds, not the 32 that running them one after the other would take. BuildKit runs every step whose inputs are ready, so independent stages give you free parallelism. In CI this is the usual layout. You keep one Dockerfile, with a checks (or lint, test) target the pipeline builds first. The release target comes afterwards, and both share the base and the cache.

The build context and .dockerignore

Before any step runs, the client sends the build context (the directory given as the last argument) to the builder. "Writing a Dockerfile" in Docker for beginners introduced .dockerignore to keep secrets out of COPY . .. It also decides how much is sent. Add the two classic offenders, a Git object store and a log file. Then compare a build with and without the ignore file:

Each build you run prints two transfers. The first (#3) is BuildKit reading .dockerignore: the file and a little metadata, and almost nothing once the file is renamed away. The second is the context. BuildKit transfers it incrementally. It sends only files that changed since the last build, plus metadata for the rest. So a rebuild of an unchanged directory moves only a few dozen bytes. With the ignore file, the new .git pack and debug.log were excluded. The context transfer stayed at that metadata-sized figure (86 bytes here). Without it, both new files went up: 35.01MB. On a laptop next to the daemon that's a fraction of a second. To a remote builder it's an upload on every build. And with --mount=type=bind,target=. the same filter decides what the build can see. The patterns follow Go's filepath.Match rules with ** support, and ! re-includes a path. A file named <Dockerfile>.dockerignore next to a Dockerfile (for example Dockerfile.single.dockerignore) overrides the shared one for builds of that Dockerfile.

Build metadata for the pipeline

Your CI jobs need the digest of what they just built, to scan it, sign it and deploy it. Don't parse it from the progress output. --metadata-file writes it as JSON:

containerimage.digest is the image index digest, the value the later pipeline stages should carry ("Tags, digests and promotion"). buildx.build.ref identifies the build record (builder/node/ID), which docker buildx history can look up later. The file also holds a summary of the provenance BuildKit recorded, and its materials list the base images by digest. "Buildx builders, outputs, cache and Bake" introduces provenance and SBOM attestations as build outputs. "Pinning, SBOMs, provenance and scanning" in Advanced container security teaches them.

Secret mounts

Some builds need a token, for a private package index or an API that serves a licence file. Such a build must not receive it as a build argument or environment variable. Those land in the image metadata or its history. "Build-time secrets and how images leak them" in Advanced container security shows each leak. The mechanism you want is a secret mount. Here you'll use a dummy token in a file:

FROM alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
RUN --mount=type=secret,id=api_token,required=true \
    echo "token file: $(wc -c < /run/secrets/api_token) bytes"
RUN --mount=type=secret,id=api_token,env=API_TOKEN,required=true \
    echo "API_TOKEN has ${#API_TOKEN} characters in this RUN"
RUN ls /run/secrets 2>&1; echo "API_TOKEN=${API_TOKEN:-<unset>}"

--secret id=api_token,src=... makes the file available to the build under the id api_token. In the first RUN it's a file at /run/secrets/api_token; target= would put it elsewhere. In the second, env=API_TOKEN exposes it as an environment variable for that one command. The third RUN has no mount, and neither the file nor the variable exists there. You should see three behaviours before you rely on this in CI:

First, the rotated token (passed from an environment variable with env=LAB_TOKEN this time) didn't re-run anything; every step is CACHED. Secret contents are deliberately left out of the cache key. So a build that must react to a new secret needs --no-cache or a changed input. Second, a build with no secret at all succeeded, because the steps came from cache and never asked for it. required=true is enforced only when the step actually runs. Third, with --no-cache the same build fails with secret api_token: not found, as it should. Finally, the image:

History records the commands, not the mount options or values, and /run/secrets doesn't exist in the image. What the step writes to its own filesystem is another matter. A RUN that copies the secret into a file, or prints it into a log the build keeps, leaks it like any other content.

SSH mounts

When you clone a private Git repository during a build, you need an SSH key, and a copied key file is a leaked key. With --ssh you forward an SSH agent instead. The build can ask the agent to sign, but it never sees the key. The lab creates a throwaway key with no passphrase and a temporary agent. Never use a real key for this exercise.

FROM alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
RUN apk add --no-cache openssh-client
RUN --mount=type=ssh \
    echo "SSH_AUTH_SOCK=$SSH_AUTH_SOCK" && ssh-add -l
RUN ssh-add -l; echo "exit status without the mount: $?"

--ssh default forwards the agent named by SSH_AUTH_SOCK in your shell. Inside the step with --mount=type=ssh, SSH_AUTH_SOCK points at a socket BuildKit provides. There ssh-add -l lists the same fingerprint as the key on the VM (fingerprints differ on every run, since the key is new). The next step, without the mount, has no agent. --ssh default=./lab_ed25519 would load an unencrypted key file without an agent. For a real clone the step also needs the server's host key, otherwise ssh refuses to connect:

RUN apk add --no-cache git openssh-client
# known_hosts holds GitHub's published host keys, committed to the repository
COPY --chmod=0644 known_hosts /root/.ssh/known_hosts
RUN --mount=type=ssh git clone git@github.com:example-org/private-lib.git /src/private-lib

Many examples run ssh-keyscan github.com in the build instead. That trusts whatever answers at build time, which is exactly the check a host key exists for. Instead, copy the keys from GitHub's published fingerprints page (docs.github.com, "GitHub's SSH key fingerprints") into a known_hosts file once. Review it like any other change, and copy it in as above.

Clean up

The docker builder prune filter removes the Go cache mount by its id, so other build cache on the VM stays. If something was left behind, /opt/secopslog-lab/setup/reset-lab.sh --images removes everything lab-related.

Try this

Before you clean up, make a second one-line edit and rebuild only Dockerfile.single. First predict whether it lands near the 11.8 seconds above or near the 1.2 seconds of the multi-stage rebuild. Then rebuild the default target after the same edit and compare. The gap you see is the Go build cache mount, since both rebuilds re-run the compile step.

Takeaway

Compile in one stage, copy only the binary into the final one, and give the compile step a cache mount. Here that took the image from 493MB to 7.74MB and a rebuild from 11.8 to 1.2 seconds. On CI runners that start cold the cache mount is empty, so measure the first build too. Never pass tokens as build arguments; and remember a cached step won't notice a rotated secret, and won't fail on a missing one until --no-cache forces it to run.

Related