Buildx builders, outputs, cache and Bake

Builders and drivers, --load and --push, cache export and Bake files.

Buildx sends every build to a builder, every builder has a driver, and the driver decides where results and cache can go. That one chain explains most build surprises in CI. You get a cache export the builder refuses, a green build that pushed nothing, or a docker buildx bake whose targets nobody can find. You'll run each piece on the lab VM against a local registry:3: builders and drivers, the outputs, cache export and import, and Bake.

Use the main lab VM, secopslog-docker, with the lesson files in ~/lab/buildx. They're a small Go module with two programs (cmd/api and cmd/worker), one Dockerfile that builds either through ARG APP, a buildkitd.toml, and a docker-bake.hcl. Nothing on the host changes. The outputs below use BuildKit's plain progress format, which is what CI logs show. In an interactive terminal you get the same steps as a collapsing view, or you can add --progress=plain.

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS build
ARG APP=api
WORKDIR /src
RUN --mount=type=bind,target=. \
    --mount=type=cache,id=lab-shop-go-build,target=/root/.cache/go-build \
    CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/app ./cmd/${APP}
 
FROM scratch
COPY --from=build /out/app /app
USER 65532:65532
ENTRYPOINT ["/app"]

Builders and drivers

A builder is a named BuildKit instance that Buildx can send builds to. Every Docker host has one called default:

The docker driver runs BuildKit inside dockerd. Its worker uses containerd and the overlayfs snapshotter, the same store the daemon's images live in. That's why builds on the default builder can use local images as bases and land in docker image ls without a copy. The GC policy lines are the default garbage-collection rules for build cache. Cache mounts and local sources are kept 48 hours within a size cap, and the rest stay within free-space limits. The docker driver has no config of its own: no buildkitd.toml and no BuildKit version choice. It builds for the host's platforms only unless emulation is installed ("Multi-platform images").

The docker-container driver runs a separate BuildKit daemon in a container, from the moby/buildkit image. Two other drivers exist for shared build infrastructure. kubernetes starts BuildKit pods in a cluster (docker buildx create --driver kubernetes --driver-opt namespace=ci,replicas=2). remote connects to a BuildKit daemon you already run (docker buildx create --driver remote tcp://buildkitd.internal:1234, normally with TLS client certificates). Both behave like docker-container in everything below.

# Pull Docker Hub images through Google's public mirror (avoids Docker Hub's
# anonymous pull limit on shared CI runners) and talk plain HTTP to the lab registry.
[registry."docker.io"]
  mirrors = ["mirror.gcr.io"]
 
[registry."localhost:5000"]
  http = true

This config does two things. First, builds pull Docker Hub images through mirror.gcr.io, Google's public cache of often-pulled Docker Hub images. That takes most pulls off a shared CI egress IP, and so away from Docker Hub's anonymous pull limit. Images the mirror doesn't hold still come from Docker Hub and count against the limit. So authenticated pulls or your own pull-through cache are still the full answer. Second, the builder talks plain HTTP to the lab registry on localhost:5000. dockerd allows that for localhost by default, but a separate BuildKit daemon needs it spelled out. Create the builder:

--bootstrap started it right away instead of on first use. The builder is the container buildx_buildkit_lab-builder0, and its state (cache, snapshots) lives in a volume next to it. --driver-opt network=host puts the container in the VM's network namespace, so localhost:5000 in the builder is the VM's port 5000. Buildx grants the matching network.host entitlement on the daemon flags. The config file was copied into the container when you created it. So if you edit your copy later, you'll need docker buildx rm and create again. The asterisk in docker buildx ls still marks default as the builder plain docker build uses.

Where the result goes

A docker-container builder has its own cache and no image store. Build without saying where the result should go:

The FROM step downloaded the whole golang image (16 seconds here) even though the daemon already had it. This builder doesn't share the daemon's store, so it pulls through its own registry config. The final warning is the point. The image exists only as cache inside the builder. You choose an output:

--load exported to a tarball ("sending tarball") and the daemon imported it. The image ID, 2cb9c78938fa, is the manifest digest shown in the export. A loaded image from this builder is a single manifest with no attestation. The default builder is different. There the image goes straight into the containerd image store and keeps its provenance attestation as an index ("Image anatomy: index, manifest, config and layers" shows one). Pushing keeps everything the builder produced. Start the lab registry and push:

It's the same manifest digest as the loaded image, plus an attestation manifest (BuildKit's default minimal provenance) and the manifest list (the index) that ties them together. The registry tag now points to the index digest. SBOM and provenance attestations are build outputs you turn on and tune with --sbom and --provenance. "Pinning, SBOMs, provenance and scanning" in Advanced container security covers what they contain and how you use them.

docker buildx use changes the builder that docker build and docker buildx build pick when you don't give --builder. It's a per-user setting in ~/.docker/buildx, and a common cause of "my build suddenly does not show up in docker images":

Here docker build ran on lab-builder and didn't warn about a missing output. That's because Buildx turns on --load by default when it's invoked as docker build. docker buildx build doesn't do that, as the earlier warning showed. Switching back with docker buildx use default keeps the rest of the course predictable. In scripts, pass --builder yourself or set BUILDX_BUILDER, rather than relying on use.

The filesystem exporters skip images entirely. Build the worker and take the binary straight out of the final stage:

type=local wrote the 1.5MB static worker binary into out/, and it runs directly on the VM. type=tar holds the same kind of filesystem (here the 5.4MB api binary) without any image metadata. type=oci is a complete image in OCI layout. index.json points at the manifest, and the blobs are the manifest, config and one layer. Tools such as Trivy, Syft and Cosign read that layout directly. Like --load, the OCI archive from this builder carries no attestation manifest.

Cache export and import

A fresh CI runner, a new builder or a pruned one starts with an empty cache. You fix that by exporting the cache to a registry and importing it on the next build. mode=max exports cache for every stage, not only the layers of the final image:

The cache lives next to the image as lab-api:buildcache, a manifest whose config type is application/vnd.buildkit.cacheconfig.v0. Its largest layer, 67,652,866 bytes, is the golang base layer. With mode=max the builder stage is in the cache, base included. Now throw the builder's local cache away and rebuild with --cache-from:

482.8MB of local cache is gone, and the rebuild still reports every step CACHED after importing cache manifest, in 2.2 seconds. BuildKit downloaded only the cache metadata. Since nothing had to run, it never fetched the golang layers either. Compare the two modes with the local cache backend, which writes the same format to a directory:

That's 2.3MB for mode=min, the layers of the final scratch image only, against 74MB for mode=max. With min, a runner whose source changed still has to pull golang and recompile. With max, it reuses every unchanged step of the builder stage, at the price of a bigger upload. Neither mode ever exports cache mounts (RUN --mount=type=cache). Other backends follow the same pattern. type=gha stores cache in the GitHub Actions cache service and only works inside a workflow. type=s3 and type=azblob use object storage, and type=inline embeds min-mode cache metadata in the pushed image.

The default builder can export cache too, as long as the daemon uses the containerd image store. That's the default on fresh Docker 29 installs and the lab VM's setup:

It's 74MB, the same size as the mode=max export from lab-builder, written by BuildKit inside dockerd. On hosts still on the classic store, any --cache-to other than inline fails on the default builder with Cache export is not supported for the docker driver. The usual fix in CI is a docker-container builder. That's what docker/setup-buildx-action creates by default, and it works whatever store the runner's daemon uses.

Bake

A real repo builds several images with the same options: platforms, labels, cache settings, tags taken from the commit. If you write those as shell flags in each pipeline, they drift. Bake reads them from a file, docker-bake.hcl, and builds the targets in parallel on one builder:

variable "REGISTRY" {
  description = "Registry the images and cache go to"
  default = "localhost:5000"
}
 
variable "TAG" {
  description = "Image tag, set by CI to the commit or release"
  default = "dev"
}
 
group "default" {
  targets = ["api", "worker"]
}
 
target "_common" {
  context    = "."
  dockerfile = "Dockerfile"
  labels = {
    "org.opencontainers.image.source" = "https://example.com/lab/shop"
  }
}
 
target "api" {
  description = "HTTP API image"
  inherits   = ["_common"]
  args       = { APP = "api" }
  tags       = ["${REGISTRY}/lab-api:${TAG}"]
  cache-from = ["type=registry,ref=${REGISTRY}/lab-api:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/lab-api:buildcache,mode=max"]
}
 
target "worker" {
  description = "Background worker image"
  inherits   = ["_common"]
  args       = { APP = "worker" }
  tags       = ["${REGISTRY}/lab-worker:${TAG}"]
  cache-from = ["type=registry,ref=${REGISTRY}/lab-worker:buildcache"]
  cache-to   = ["type=registry,ref=${REGISTRY}/lab-worker:buildcache,mode=max"]
}

Variables have defaults, and environment variables of the same name override them. _common holds the shared settings. api and worker inherit it and add their build argument, tags and cache refs. The group default is what docker buildx bake builds when you name no target. Ask Bake what it sees before you build anything:

--print shows the fully resolved definition as JSON, with inheritance applied and ${REGISTRY}, ${TAG} filled in. It's the first thing to run when a CI build does something odd. _common isn't listed as a target because names starting with _ are treated as internal. Build and push both images with a tag from the environment:

Both targets ran in one build session. The ERROR for the worker cache is the first build's normal state. Nothing has been exported to lab-worker:buildcache yet, so the import fails and BuildKit builds without it. The summary at the end repeats the failed import without failing the build. The api import worked from the cache pushed earlier. The registry now has both repos, and lab-worker:2 is an index with its attestation, like any pushed build.

In CI you can change a definition without editing the file. Use --set target.key=value (with * for every target), and set variables through the environment:

The worker now builds for two platforms with tag 3. --print shows the value as one string, linux/amd64,linux/arm64, exactly as it was set. Buildx splits the comma list into two platforms when it builds. A GitHub Actions job is usually only this much, with the bake file doing the rest. Each action is pinned to a full commit SHA with its version in a comment, for the reason "Docker in CI/CD: build to promote" gives. The job token gets read access only:

permissions:
  contents: read
 
jobs:
  images:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
      - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
        with:
          registry: registry.example.com
          username: ${{ vars.REGISTRY_USER }}
          password: ${{ secrets.REGISTRY_TOKEN }}
      - uses: docker/bake-action@018cb6412ab401ebaa809aa5f85966b74628600f # v7.4.0
        env:
          REGISTRY: registry.example.com/shop
          TAG: ${{ github.sha }}
        with:
          push: true
          set: |
            *.cache-from=type=gha
            *.cache-to=type=gha,mode=max

setup-buildx-action creates a docker-container builder on the runner, so you get every output and cache backend above. The --set lines swap the registry cache for the GitHub cache without touching the file developers use locally. "Docker in CI/CD: build to promote" puts this build into the full pipeline with scanning, signing and promotion.

Clean up

docker buildx rm stops the builder container and deletes its state volume with the cache in it. (--keep-state keeps the volume for a later builder of the same name.) rm -v on the registry removes its anonymous storage volume with the pushed images. The moby/buildkit image stays in the image store for the next builder, and reset-lab.sh removes any lab- builders you forget.

Your turn

Rerun the bake-set step with TAG=4 and a different --set key, for example *.platform=linux/arm64 alone, and predict what --print will show in platforms and tags before you look. Compare with the output above, where one comma-joined string came back. Then run docker buildx bake --list=targets again and check that _common still doesn't appear as a target.

Takeaway

Always name where a build goes: --builder or BUILDX_BUILDER for the builder, and --load, --push or -o for the result. Without that, docker buildx build on lab-builder left the image only as cache, and a stray docker buildx use quietly moved plain docker build to another builder. For CI caches, mode=max cost 74MB against 2.3MB for mode=min here, but it's what let a pruned builder rebuild every step CACHED in 2.2 seconds.

Related