Minimal bases: distroless, scratch, static and Alpine

Choose a runtime base and still be able to debug it.

docker exec lab-probe sh fails with exec: "sh": executable file not found in $PATH. On a distroless image that's the design working. An attacker who gets code execution finds no shell, no package manager and no downloader to reach for, and there's no apt-get install to bring them back. The cost is that the image also lacks the small runtime files a program quietly expects, such as CA certificates, a timezone database or a user entry. Debugging needs a different method too. You'll build one Go program on three bases to see exactly which files matter, and prove static versus dynamic linking. Then you'll compare Alpine and Debian and debug the shell-less container from a sidecar.

Use the main lab VM. The lesson files go in ~/lab/distroless. The distroless images come from gcr.io, not Docker Hub, so they don't count against Docker Hub's pull limit.

What a distroless image contains

Google's distroless project publishes images built from Debian 13 packages: static-debian13 for programs that need no C library, base-debian13 which adds glibc and OpenSSL, cc-debian13 which adds the C++ runtime, and language images such as python3-debian13, nodejs24-debian13 and java21-debian13. Each comes in four tags: latest (runs as root), nonroot (UID 65532), and debug or debug-nonroot, which add a BusyBox shell. Pull the ones you need here and read the config of the static image. (The loop at the end pulls alpine:3.24 and debian:trixie-slim for the size comparison if an earlier lesson hasn't.)

The config sets a user, an environment and a working directory, and no Cmd or Entrypoint; your image supplies those. That's why docker create on the bare base fails with no command specified. To look inside such an image, give docker create any command at all; it is never run:

The 1400 entries include a few that docker export adds for every container (.dockerenv, etc/hosts, etc/resolv.conf). Of the rest, 1247 are timezone files. There are no ELF executables at all, no shell and no package manager. What is there is exactly the runtime support a static program needs: a passwd file with root, nobody and nonroot, and a CA certificate bundle. The same check, with the grep for shells and package managers, makes a useful CI gate on your own final images.

One program on three bases

probe is a small Go program that reports its user and tries the four things that typically break on a minimal base. Those are a user lookup, a timezone conversion, a DNS lookup and an HTTPS request. Run with serve, it's an HTTP server on port 8080 for the debugging section.

// probe: checks the runtime files a minimal image may or may not have.
package main
 
import (
	"fmt"
	"net"
	"net/http"
	"os"
	"os/user"
	"time"
)
 
func check(name string, err error, ok string) {
	if err != nil {
		fmt.Printf("%-5s FAIL  %v\n", name, err)
		return
	}
	fmt.Printf("%-5s ok    %s\n", name, ok)
}
 
func main() {
	if len(os.Args) > 1 && os.Args[1] == "serve" {
		http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "probe ok") })
		fmt.Println("listening on :8080")
		fmt.Println(http.ListenAndServe(":8080", nil))
		return
	}
	fmt.Printf("uid=%d gid=%d\n", os.Getuid(), os.Getgid())
 
	u, err := user.Current()
	check("user", err, func() string { if u != nil { return u.Username + " " + u.HomeDir }; return "" }())
 
	loc, err := time.LoadLocation("Europe/Berlin")
	check("tz", err, func() string { if loc != nil { return time.Date(2026, 7, 1, 12, 0, 0, 0, time.UTC).In(loc).Format("15:04 MST") }; return "" }())
 
	addrs, err := net.LookupHost("mirror.gcr.io")
	check("dns", err, fmt.Sprintf("%d addresses", len(addrs)))
 
	c := &http.Client{Timeout: 10 * time.Second}
	resp, err := c.Get("https://mirror.gcr.io/v2/")
	if err == nil {
		resp.Body.Close()
	}
	check("tls", err, func() string { if resp != nil { return resp.Status }; return "" }())
}

The Dockerfile builds it once with CGO_ENABLED=0 and copies the binary onto three bases: plain scratch, scratch plus three runtime files, and distroless static. Note the numeric USER on the scratch stages. The builder and the distroless base are pinned by digest, so a rebuild next month starts from the same files. A bot moves the pins, as "Pinning, SBOMs, provenance and scanning" describes. Dockerfile comments must start a line; a comment after FROM, COPY or USER breaks the instruction.

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS build
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/probe .
RUN echo 'app:x:10001:10001:app:/nonexistent:/sbin/nologin' > /out/passwd
 
FROM scratch AS scratch-bare
COPY --from=build /out/probe /probe
USER 10001:10001
ENTRYPOINT ["/probe"]
 
FROM scratch AS scratch-full
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /usr/local/go/lib/time/zoneinfo.zip /usr/share/zoneinfo.zip
ENV ZONEINFO=/usr/share/zoneinfo.zip
COPY --from=build /out/passwd /etc/passwd
COPY --from=build /out/probe /probe
USER 10001:10001
ENTRYPOINT ["/probe"]
 
FROM gcr.io/distroless/static-debian13:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 AS distroless
COPY --from=build /out/probe /probe
ENTRYPOINT ["/probe"]

On bare scratch three of the four checks fail, each for a missing file:

scratch-full copies those three things from the build stage and passes all four checks. Its HTTPS request gets 401 Unauthorized from mirror.gcr.io, an HTTP answer that only arrives after the certificate verified. Distroless static passes all four the same way, because it ships those files. The next check shows the other reason for a numeric USER: Docker resolves a user name against the image's /etc/passwd when the container starts, and on scratch there's no file to resolve against.

A numeric UID needs no entry. That's also why distroless/static:nonroot and the scratch stages above work with volume ownership set by number; "Run as non-root" covers that part.

Static or dynamic: prove it

Go binaries are static only when cgo is off. Go enables cgo by default for native builds when it finds a C compiler. golang:*-alpine has none, so cgo was off above. The Debian-based golang images include gcc, and they produce binaries linked against glibc for code that uses net or os/user. Dockerfile.cgo turns cgo on deliberately (after installing gcc and the musl headers) and puts the result on distroless base, which does have a C library, but glibc:

FROM golang:1.27-alpine@sha256:b8c9b82affbbb82a020bbcf998dad2ba9516944ac7e883524642a481a9505e9c AS build
RUN apk add --no-cache gcc musl-dev
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=1 go build -trimpath -o /out/probe .
 
FROM gcr.io/distroless/base-debian13:nonroot@sha256:a0d70d6a97cd697d9362bc2aae4a6560dd65817e365d0043b07325a97975dc91
COPY --from=build /out/probe /probe
ENTRYPOINT ["/probe"]

no such file or directory names /probe, which is plainly there. The missing file is the program interpreter, the dynamic loader the kernel must start first. file and ldd on the two binaries show it:

The static build is statically linked and ldd says not a dynamic executable. The cgo build names its interpreter, /lib/ld-musl-aarch64.so.1, and needs libc.musl-aarch64.so.1. The VM's glibc ldd substitutes its own loader for that interpreter line and can't find the musl C library at all. A distroless or Debian base has glibc's loader, not musl's. So a binary linked on Alpine won't run there, and the reverse is equally true. On an amd64 machine the names read x86-64, /lib/ld-musl-x86_64.so.1 and libc.musl-x86_64.so.1. Put file on your build artifact into CI and fail on dynamically linked when the target base is scratch or static.

Building C with gcc -static against glibc doesn't give the same guarantee. glibc's user and host lookups (getpwnam, getaddrinfo) go through the Name Service Switch. It loads libnss_* modules at run time even in a statically linked program. So the binary starts on scratch and then fails on its first lookup. The linker warns about this when it links those functions statically. For static C or Rust, link against musl (for Rust the x86_64-unknown-linux-musl or aarch64-unknown-linux-musl target); for Go, CGO_ENABLED=0.

Alpine, musl and Debian slim

These are disk-usage figures on this VM (unpacked layers plus stored content, as explained in "Layers and the build cache" in Docker for beginners). alpine:3.24 is a tenth of debian:trixie-slim, and distroless static is smaller still. Size isn't the whole decision. Alpine uses musl instead of glibc, and the differences show up in production:

The musl behaviour is documented on the musl wiki; the binary incompatibility you saw in the lab. Choose Alpine for static or musl-native programs and for small utility images. Choose Debian slim (or distroless base or cc) when the service pulls in native extensions, vendor binaries or deep recursion in threads. The same goes when a DNS difference would be expensive to discover in production. For a static Go or Rust binary, distroless static or scratch with the three files above beats both.

Debugging a container without a shell

Start the probe as a server. docker exec has nothing to run:

Bring the tools to the container instead. A debug container started with --pid container:NAME joins the target's PID namespace, and with --network container:NAME its network namespace. Its tools then see the target's processes and its localhost, while the target's image stays untouched:

PID 1 is /probe serve running as 65532, the listener on 8080 is visible, and wget reaches the app over the shared loopback. Reading the target's filesystem through /proc/1/root needs more: the kernel requires ptrace access to another process's root, and Docker's default capability set doesn't include CAP_SYS_PTRACE. Grant it to the debug container only, for the session:

A debug container sharing the target's PID namespace can also signal and trace the target. So treat it like an interactive root session: start it with --rm, keep it short, and use an image you trust (busybox:1.37 here). On Kubernetes, kubectl debug creates the same kind of ephemeral container. docker debug exists too, but it's a Docker Desktop feature and not part of Docker Engine. The basic no-shell tools (inspect, logs, cp) are in "Inspect, logs and exec" in Docker for beginners.

The :debug images are for reproducing a problem on your own machine:

It's the same base with a BusyBox of 382 applets under /busybox, outside the usual PATH directories. A :debug tag that ships to production undoes the point of the base. The export check from the first section catches that, because it matches busybox and sh by name, wherever they are.

Clean up

The distroless base images stay in the image store for the next lessons; remove them with docker image rm if you need the space.

Try this

On the main lab VM, repeat the --user app step against lab-min:scratch-full and lab-min:distroless instead of scratch-bare. Predict which of them can resolve the name app, keeping in mind that distroless ships a passwd with root, nobody and nonroot only. Then run lab-min:distroless with --user nonroot and compare its user line with the earlier distroless run. Each container uses --rm, so there's nothing to clean up.

Takeaway

Ship a shell-less final image, and gate it in CI with file on the binary plus the export grep for shells and package managers. In the lab the cgo build died with no such file or directory while /probe was plainly there, an error that sends people hunting for the wrong file when the loader is what's missing. When you need tools, bring a short-lived --pid container: sidecar rather than a :debug tag that can slip into production.

Related