Multi-platform images

linux/amd64 and linux/arm64 from one build: emulation, cross-compiling and indexes.

The lab VM is arm64. On it the learner ubuntu isn't in the docker group, so you'll use sudo docker, and the lesson files go into ~/lab/multiplatform. Pull the local registry image and the native Go image up front; the builds fetch everything else themselves. Then ask the VM to build an amd64 image, and the first RUN fails:

exec /bin/sh: exec format error is the kernel refusing to run an x86-64 binary on an Arm CPU. BuildKit resolved the linux/amd64 entry of the alpine index and unpacked it, but it couldn't run a single program in it. docker buildx ls lists linux/arm64 as the only platform. binfmt_misc has no handler for x86-64 (the python3.14 entry is Ubuntu's handler for Python bytecode). You'll hit the same wall on an amd64 CI runner building for arm64 Graviton or Apple Silicon users. There are three ways around it: emulate the other CPU, cross-compile on the native one, or send each platform to a machine that has it. You'll run the first two here and see the third in config.

If you're on an amd64 machine, swap the platforms throughout: install the arm64 emulator and expect x86_64 where this page shows aarch64.

Emulation with QEMU and binfmt_misc

binfmt_misc is a kernel feature that maps a binary format, recognised by magic bytes, to an interpreter. Register QEMU's user-mode emulator for x86-64 ELF files, and the kernel runs every amd64 program through it. That includes programs started inside containers. The tonistiigi/binfmt image, the method Docker's docs use, carries static QEMU builds and registers them. It needs --privileged because it writes to /proc/sys/fs/binfmt_misc, a host-wide kernel setting. That's why you're using the sec VM. The image is pinned by tag and digest:

The first lines are the pull of the pinned image. Then comes installing: amd64 OK, and the supported list now includes linux/amd64 and linux/amd64/v2. The handler file shows the interpreter path and the flags POCF. F (fix binary) matters most. The kernel opens the QEMU binary once at registration, so it works inside containers whose filesystem has no /usr/bin/qemu-x86_64. The registration lives in kernel memory and disappears at reboot unless something registers it again. Docker Desktop does this for you. On Linux, rerun the container at boot, or install the distribution's QEMU user-mode binfmt package, which registers the handlers at boot through systemd-binfmt. On Ubuntu 26.04 that's qemu-user-binfmt, and the older qemu-user-static name has no installable package there.

This time docker buildx ls listed linux/amd64 (+2) at once, the amd64 baseline plus two microarchitecture levels. On Engine 29.8.2 the list changed only after a dockerd restart. The restart step shows the same list. On an older engine, restart dockerd if the new platform doesn't appear. Builds work either way, because the kernel does the emulation. Now build both platforms in one command and load the result:

FROM alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
RUN echo "RUN executed on $(uname -m)" > /built-on.txt
CMD ["sh", "-c", "cat /built-on.txt; echo \"container runs on $(uname -m)\""]

BuildKit runs the stages once per platform (the [linux/amd64 ...] and [linux/arm64 ...] steps). It exports a manifest and an attestation per platform, then one manifest list. --load put the whole index into the local store, and --tree shows both platforms with content. --platform on docker run picks the entry. The amd64 container reports x86_64 both at build time and at run time, because QEMU translates every instruction. That's the strength of emulation, no Dockerfile changes, and also its cost.

Cross-compiling instead of emulating

Emulation is fine for apk add and small scripts. But a compiler under QEMU runs every instruction through a translator. Compare the same Go build both ways. Dockerfile.emulated is an ordinary Dockerfile. The cross-compiling one starts the build stage on the build machine's own platform and asks the compiler for the target:

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS build
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=0 \
    go build -trimpath -ldflags="-s -w -X main.compiledOn=$(uname -m)" -o /out/lab-app .
 
FROM scratch
COPY --from=build /out/lab-app /lab-app
USER 65532:65532
ENTRYPOINT ["/lab-app"]

FROM --platform=$BUILDPLATFORM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -trimpath -ldflags="-s -w -X main.compiledOn=$(uname -m)" -o /out/lab-app .
 
FROM scratch
COPY --from=build /out/lab-app /lab-app
USER 65532:65532
ENTRYPOINT ["/lab-app"]

BuildKit defines these build arguments for you. BUILDPLATFORM is the machine running the build, here linux/arm64. TARGETPLATFORM is the platform being built, such as linux/amd64. Their parts are TARGETOS, TARGETARCH and TARGETVARIANT, and there are BUILD* equivalents too. FROM --platform=$BUILDPLATFORM pins the build stage to the native golang image, so the compiler runs at full speed, and GOOS/GOARCH make it emit code for the target. The final FROM scratch stage has no --platform, so it's the target platform's stage, and the binary is copied into it. Declare the ARGs in the stage that uses them, or they're empty.

The binary prints the CPU of the machine that ran the compiler (uname -m at build time) and the platform it runs as. The emulated build compiled on x86_64, so inside QEMU, and the cross build compiled on aarch64, natively. Both produce an amd64 program. It took 51 seconds against 5. The emulated figure includes downloading the amd64 golang image, which an emulated build stage needs and a cross build doesn't. So this small program overstates the ratio. On real projects with many packages the emulated compile itself dominates, and the gap grows to minutes. Your times depend on your CPU and connection. That gap is why cross-compiling is the default choice for Go, Rust and other toolchains that support it: CI minutes, and builds that stop timing out. The catch is language-specific. C code needs a cross toolchain (for Alpine and Debian, helpers such as tonistiigi/xx provide the wrappers). And tools that run what they just built still need the target CPU.

Native nodes

The third option avoids both emulation and cross toolchains. You give one builder several nodes, each on a machine with the right CPU, and BuildKit sends each platform to the node that supports it. The nodes are usually docker-container builders on remote Docker hosts (over SSH or a Docker context) or BuildKit pods through the kubernetes driver:

docker buildx create --name multi --driver docker-container \
  --platform linux/amd64 ssh://ci@amd64-builder.internal
docker buildx create --name multi --append --driver docker-container \
  --platform linux/arm64 ssh://ci@arm64-builder.internal
docker buildx build --builder multi --platform linux/amd64,linux/arm64 \
  --push -t registry.example.com/shop/api:1.8 .

--append adds a node to an existing builder. Native nodes give native speed for every platform, but you have to run and secure the build machines. Managed services such as Docker Build Cloud sell the same model. Many teams mix approaches: cross-compile where the toolchain allows it, and use QEMU for the few RUN steps that must run target binaries.

Pushing an index

The usual result of a multi-platform build is a push. Start the lab registry and build for both platforms with the cross-compiling Dockerfile. It's the command a release job runs, with your registry in place of localhost:5000:

Step names such as [linux/arm64->amd64 build 4/4] mark a stage running on arm64 for an amd64 target, the cross-compile. Those steps were cached from the earlier build, and only the arm64 compile ran. docker buildx imagetools inspect reads the index from the registry. It holds one manifest for linux/amd64, one for linux/arm64, and two unknown/unknown attestation manifests, each pointing at its image through vnd.docker.reference.digest. "Image anatomy: index, manifest, config and layers" took this structure apart. Promote and pin the top-level index digest, as in "Tags, digests and promotion". Each client then picks its own entry:

Without --platform, the arm64 VM got the arm64 entry. With --platform linux/amd64 it got the amd64 binary, and it ran because the emulator's still registered. On a host without emulation that second run fails with exec format error, the same failure you saw at the start.

--load and the image store

Whether you can load a multi-platform result into the local image store depends on the store. On a fresh Docker 29 install it's the containerd image store, which keeps indexes and per-platform content:

Upgraded hosts often still run the classic graph-driver store ("Where Docker keeps data" explains both). It stores one image per name, with no index. Remove the containerd-store images first, because switching stores hides them. Then switch this VM to the classic store with the steps from "Configuring the daemon safely". Back up daemon.json (or record {} when there's none), merge the key into the backup with jq, validate and restart. Try the same build, first on the default builder and then on a docker-container builder. The second build pipes through tail, so echo "exit=${PIPESTATUS[0]}" prints the build's own exit status:

Both builders refuse with docker exporter does not currently support exporting manifest lists. The build itself would work, but the classic store has nowhere to put an index. A single platform loads fine. On such hosts you've got four choices. Load one platform at a time for local testing, or push the multi-platform result to a registry (the usual CI path, which never needed --load). You can also export with -o type=oci, or migrate the host to the containerd image store.

Switching back needs an explicit setting. Restoring the backup isn't enough on this VM. Once /var/lib/docker holds classic-store data, Docker 29 keeps using the classic store on the next start. So merge the feature set to true into the backup instead, then remove the emulator:

--uninstall qemu-x86_64 removes the handler, and binfmt_misc is back to the Python entry it started with. The classic-store data from the test is still under /var/lib/docker, which is why the features line stays in daemon.json. On this disposable VM that doesn't matter, and ./setup/create-lab.sh --profile sec --recreate starts clean.

Try this

On secopslog-docker-sec, right after the compare step, rebuild lab-app:cross with the step 12 command but --platform linux/arm64, under time as before. Run it with sudo docker run --rm --platform linux/arm64 lab-app:cross and predict both halves of the line it prints. Then compare with the first pull-run line above, which came from the same Dockerfile.

Takeaway

Cross-compile when the toolchain allows it, and keep QEMU for the RUN steps that must run target binaries. The emulated Go build took 51 seconds against 5 for the cross build here, and that gap grows on real projects. On CI hosts that still use the classic store, plan for docker exporter does not currently support exporting manifest lists: push the index to a registry instead of reaching for --load.

Related