Tags, digests and promotion

Mutable tags, immutable digests and promoting by digest.

QA signs off on app:1.0 in staging at two in the afternoon. At three, a developer rebuilds a quick fix and pushes it under the same tag, because the version number "has not shipped yet". At four, the release job copies app:1.0 from staging to production. Production now runs an image nobody tested, and every dashboard says 1.0. Nothing in that story is a bug in Docker. A tag is a name that anyone with push rights can move. The fix is to record and promote the digest, which can't move. You'll reproduce the incident with a local registry, then do the promotion properly. Everything runs on the main lab VM, secopslog-docker, with the lesson files (a Dockerfile and a small check script) in ~/lab/tagsdigests.

Pinning the base image

Start where most images start, the FROM line. The lab Dockerfile pins alpine by tag and digest:

# alpine:3.24, pinned to the index digest the tag pointed at when this lab was written
FROM alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
ARG BUILD=1
RUN echo "lab-app build $BUILD" > /build.txt
CMD ["cat", "/build.txt"]

In name:tag@sha256:... the digest decides what's used. The tag is only there for people reading the file. Compare the pin with what the tag points at today:

--format '{{.Manifest.Digest}}' prints the digest of whatever the tag resolves to. For alpine that's its index ("Image anatomy: index, manifest, config and layers" explains the index). When this lab ran, the two values matched. Next time Alpine rebuilds 3.24 for a security fix, the tag moves and the lines differ. Your builds keep using the pinned bytes until someone changes the pin on purpose. "Pinning, SBOMs, provenance and scanning" in Advanced container security shows you how to move pins forward on a schedule with Renovate or Dependabot.

Pin the index digest, the one the tag resolves to. Each builder then picks its own platform from the index. So the same line works on an amd64 laptop and an arm64 CI runner. The digest of one platform's manifest (an entry inside the index) pins that architecture only. An arm64 builder given an amd64 manifest digest builds for amd64 or fails. Use a platform digest only when you mean to limit it that way, and write the reason next to it.

One tag, two builds

Start a registry:3 container on a lab network, published on the VM's loopback address. Docker trusts localhost registries over plain HTTP, so you don't need any TLS setup for the lab:

Build the image, push it to the staging repository, and record its digest. This is the build QA tests:

The push ends with 1.0: digest: sha256:..., and the lookup prints the same value. It's the index digest, because BuildKit attached a provenance attestation and made the result an index. tee saves it in tested.digest. That file is your release record. Now the afternoon fix arrives under the same tag:

Same tag, new digest. The registry took the overwrite and said nothing. Only the alpine layer and the empty attestation config were reused. Anyone who pulls 1.0 from now on gets build 2. Anyone holding the digest still gets build 1:

The third command uses name:tag@digest. The tag says 1.0, which is build 2, and the container still prints build 1. When a reference carries a digest, Docker resolves the digest and ignores the tag. That's how your Compose files and Kubernetes manifests should name production images. Most of the layers were already present, so the pull finished almost at once. Build 1's content was still in the local content store; only the tag had moved off it.

Image ID, RepoDigests and platform digests

Deploy scripts often compare "the image we meant" with "the image we got". So you need to know which field holds what:

On the containerd image store, the default on fresh Docker 29 installs, the image ID is the index digest. So the ID and the registry digest agree, as here. On hosts still using the legacy graph-driver store, the ID is the digest of the image config, which no registry reports ("Inspecting, exporting and cleaning up images" shows it). RepoDigests is the portable field. It holds repository@digest for every registry this image is known to have come from, on both stores. A running container records the ID in docker inspect --format '{{.Image}}' CONTAINER. If your fleet mixes old and new hosts, compare digests from the registry or from RepoDigests, not raw IDs.

The digest you recorded names an index. Inside it are the platform manifest and its attestation:

On this arm64 VM the only runnable entry is linux/arm64. Built on amd64 it'd be linux/amd64, and a multi-platform build ("Multi-platform images") lists both. Record and promote the index digest, never the inner manifest digest. Otherwise the attestation and any other platforms are left behind.

Promotion by digest

The lab registry has two repositories standing in for two environments: lab-staging/app and lab-prod/app. In a real setup they're often two registries or projects, each with its own credentials. Promotion copies the tested digest to production. It never rebuilds. A rebuild from the same commit produces a different digest anyway, because the config records the build time. So "the same source" isn't "the same image". docker buildx imagetools create copies an image between repositories registry-side, without pulling it into the local store:

The two copying lines are the manifests the index lists, the arm64 image and its attestation, each with the blobs it references. The last line pushes the index itself under the new tag. The image never passed through the local store. The check script turns "is production running what we tested" into an exit status your pipeline can act on:

#!/bin/sh
# Usage: ./check-digest.sh IMAGE:TAG DIGEST_FILE
# Exits 0 only when IMAGE:TAG currently resolves to the digest recorded in DIGEST_FILE.
want=$(cat "$2")
have=$(docker buildx imagetools inspect "$1" --format '{{.Manifest.Digest}}')
if [ "$have" = "$want" ]; then
  echo "OK    $1 -> $have"
else
  echo "FAIL  $1 -> $have"
  echo "      tested image is $want"
  exit 1
fi

A reachable registry is not an approval

Promotion by digest only keeps you safe if it's the one way into production. The registry has no idea which images were tested. Anyone who can reach it with push rights can overwrite the production tag, here with the untested build 2:

The push succeeded and production silently changed. Only the check noticed. What matters is the order of controls, not the script. Give production push rights to the promotion job alone. Turn on immutable tags where the registry offers them (Harbor, Amazon ECR, Azure Container Registry and Google Artifact Registry do; registry:3 doesn't). Deploy by digest, so a moved tag can't change a running service. And verify a signature made over the tested digest before anything runs. "Signing, verifying and promoting by digest" in Advanced container security covers that, and "Docker in CI/CD: build to promote" shows the full pipeline. Being able to reach a registry is a path in, not a sign-off.

Put production back on the tested image. The plain docker tag and docker push route works too, as long as the source is the digest and not the tag:

Nothing was uploaded, since the repository already had every blob. The pushed digest equals the tested one, attestation included. With the containerd store, pushing a pulled image keeps its index intact, but you should check the printed digest anyway. A host on the legacy store keeps only the platform image, not the index. So a re-push from there can't reproduce the original digest. imagetools create avoids the question by never going through the local store. Clean up:

Your turn

Before the cleanup, point the check script at staging instead: sh check-digest.sh localhost:5000/lab-staging/app:1.0 tested.digest. Predict whether it prints OK or FAIL and which exit status you'll get, then compare with the check2 output above. Staging's 1.0 still names the afternoon rebuild, so the tag you'd naively promote is the wrong one.

Bottom line

Deploy and promote by the index digest, never by the tag: the bypass push overwrote lab-prod/app:1.0 with build 2 and the registry said nothing. A digest check only catches that if it runs after every push, and only immutable tags plus push rights limited to the promotion job stop it from happening. Rebuilding from the same commit is no substitute, because the new config gets a new digest.

Related