Docker in CI/CD: build to promote

Build, test, scan, push, attest, sign, verify and promote by digest.

Intermediate18 min · lesson 24 of 24
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (11 files, 5 KB): dockerci.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/docker-hard/dockerci.tar.gz && tar -xzf dockerci.tar.gz, which creates ~/lab/dockerci/. SHA-256: cd55552d91364ab91e7425b67e32c41fc20c2f40bcbe05c722ec139e90bb5267

A common release job builds the image twice. It builds once with load: true so Trivy can scan it on the runner, and once with push: true for the registry. The scan passes and the push succeeds, but the image in the registry isn't the one that was scanned. It's a second build. It matches the first only if every layer came from cache, and with provenance attached its digest differs anyway. Then a cosign sign step signs whatever the tag points at by the time it runs. Each step looks right; the order and the identity of the artifact are wrong. You'll fix that here. You'll build a pipeline in which one image, named by one digest, goes through every gate. You can't promote it to production without a signature over that digest. You'll run it on the main lab VM, secopslog-docker, in ~/lab/dockerci with the lesson files. It's a script you could run in any CI system. At the end you'll see the same steps as a GitHub Actions workflow.

The order

One commit, one digest, nine steps
1Build
once, with provenance
2Test
run that image
3Scan
fail on fixable HIGH/CRITICAL
4Push
to the build repository
5Resolve digest
must equal the build's
6Attest
SBOM, signed
7Sign
the digest
8Verify
signature + attestation
9Promote
by digest: staging, then prod
Nothing reaches a registry before test and scan pass. Signing needs a digest that exists in a registry, so it comes after the push. Each promotion verifies first.

The order follows from what each step needs. Test and scan must run on the image you'll ship. So you build once, and each later step uses that output. The push comes after the gates, so a failing build never shows up in any registry. Signing and attesting come after the push. Cosign signs a manifest digest, and it stores the signature in the registry next to it. Before the push there's nothing to sign. The digest is checked against the build's own record, so the signature covers the bytes you tested. Verifying is its own step, and a different job repeats it before each promotion.

Three repos carry the result, and a push to one isn't an approval:

Anyone with push rights to a registry can put an image there ("Tags, digests and promotion" showed a direct push overwriting a production tag). A digest is approved only when its signature and attestation verify against your key or your CI identity. Put that policy in the promotion job. In Kubernetes, also put it in an admission controller that verifies before it admits a pod.

Tools, keys and registries

The lab uses the same pinned tools as two lessons in Advanced container security: "Pinning, SBOMs, provenance and scanning" and "Signing, verifying and promoting by digest". Those lessons cover Cosign and Syft in depth. The tools are Cosign 3.1.4, Syft 1.55.0 and Trivy 0.76.1, downloaded from their GitHub releases into ~/lab/bin. Nothing's installed system-wide. The script that fetches them carries its own SHA-256 values:

install-tools.sh
#!/bin/sh
# Usage: ./install-tools.sh TOOL... (TOOL: cosign syft trivy grype)
# Downloads pinned release binaries from GitHub, checks each against the SHA-256 committed below,
# and installs them into $BIN (default ~/lab/bin). Nothing system-wide.
# The hashes live in this file, in your repository: a release asset replaced on GitHub after you
# pinned it fails the check. Bumping a version means updating its two hashes in a reviewed change.
set -eu
BIN=${BIN:-$HOME/lab/bin}
COSIGN=3.1.4 SYFT=1.55.0 TRIVY=0.76.1 GRYPE=0.121.0
case $(uname -m) in
aarch64|arm64) A=arm64 T=ARM64 ;;
x86_64) A=amd64 T=64bit ;;
*) echo "unsupported CPU $(uname -m)" >&2; exit 1 ;;
esac
sha() { # sha ASSET: the pinned SHA-256 of a release asset
case $1 in
cosign-linux-arm64) echo ade83b7aac62e131cacfe96d958948b2150cc9efec46a570a2649b8e27dedad2 ;;
cosign-linux-amd64) echo 424e2c97d0b45fd7a6b608a87f090f35633f4c23ece5f092d48bba11aaf4bbfa ;;
syft_1.55.0_linux_arm64.tar.gz) echo 929bd163db2d7cd2255d3bcb8967208079813d7b46b0c952f7b794d8e2c7d304 ;;
syft_1.55.0_linux_amd64.tar.gz) echo bb65249542db5fedeeb4d3f9b018b75e7264dbfbbb10486f044581a92f90490d ;;
trivy_0.76.1_Linux-ARM64.tar.gz) echo e174dff2a6dba5e3ae41f8d0537a6324405bd08132497340304c6b20e0f16a1f ;;
trivy_0.76.1_Linux-64bit.tar.gz) echo c7cbe3137fb8a48a17574bede3dd6beb16742d1e9772929940fc90082c39d9c0 ;;
grype_0.121.0_linux_arm64.tar.gz) echo 86f570f75ca947ea3dd6486309232bcff6d78e44ba081ae75ab14eab72be3dae ;;
grype_0.121.0_linux_amd64.tar.gz) echo 64e07133d3486115de647a66581461dccf5c732748a0a3119077395565f15519 ;;
*) echo "no pinned hash for $1" >&2; return 1 ;;
esac
}
GH=https://github.com
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
mkdir -p "$BIN"
fetch() { # fetch URL_DIR ASSET: download the asset and check it against the pinned hash
want=$(sha "$2")
curl -fsSL --retry 10 --retry-delay 5 --retry-all-errors -o "$tmp/$2" "$1/$2"
(cd "$tmp" && echo "$want $2" | sha256sum -c -)
}
for t in "$@"; do
case $t in
cosign) u=$GH/sigstore/cosign/releases/download/v$COSIGN
fetch "$u" "cosign-linux-$A"
install -m 0755 "$tmp/cosign-linux-$A" "$BIN/cosign" ;;
syft) u=$GH/anchore/syft/releases/download/v$SYFT
fetch "$u" "syft_${SYFT}_linux_$A.tar.gz"
tar -xzf "$tmp/syft_${SYFT}_linux_$A.tar.gz" -C "$BIN" syft ;;
trivy) u=$GH/aquasecurity/trivy/releases/download/v$TRIVY
fetch "$u" "trivy_${TRIVY}_Linux-$T.tar.gz"
tar -xzf "$tmp/trivy_${TRIVY}_Linux-$T.tar.gz" -C "$BIN" trivy ;;
grype) u=$GH/anchore/grype/releases/download/v$GRYPE
fetch "$u" "grype_${GRYPE}_linux_$A.tar.gz"
tar -xzf "$tmp/grype_${GRYPE}_linux_$A.tar.gz" -C "$BIN" grype ;;
*) echo "unknown tool $t" >&2; exit 1 ;;
esac
done

A checksum file downloaded from the same release as the binary only catches a damaged download. Whoever can replace the binary on the release page can replace the checksum file next to it. Hashes you commit to your repo, taken when you reviewed the version, catch that too. It's the same idea as the hashed lock file in "Recipe: a Python web app", which protects the Python dependencies. Moving to a new tool version then becomes a reviewed change of the version and its two hashes. On a flaky connection, as in this run, curl prints curl: (35) lines while it retries (--retry 10 in fetch). The OK lines prove what arrived.

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ ./install-tools.sh cosign syft trivy export PATH="$HOME/lab/bin:$PATH" cosign version --json | jq -r '"cosign \(.gitVersion)"'; syft version | grep '^Version'; trivy --version | head -1
curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer cosign-linux-arm64: OK curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer curl: (35) Recv failure: Connection reset by peer syft_1.55.0_linux_arm64.tar.gz: OK trivy_0.76.1_Linux-ARM64.tar.gz: OK cosign v3.1.4 Version: 1.55.0 Version: 0.76.1
$ export COSIGN_PASSWORD=lab-only-test-password cosign generate-key-pair cosign signing-config create --out lab-signing-config.json
Private key written to cosign.key Public key written to cosign.pub
$ docker run -d --name lab-registry -p 127.0.0.1:5000:5000 registry:3 docker run -d --name lab-prod-registry -p 127.0.0.1:5001:5000 registry:3
05eecda9f63ac9f8aea32ea136801868a1c12bb0459b4a5140efae587c776dc8 83c822d02591e411412b2fe36a43d85b9ffa85298b7010ce7a249b2e43dfba2a

Locally, you sign with a key pair protected by a test password (COSIGN_PASSWORD, exported in the shell). You also use a signing config with no transparency log, so lab signatures aren't published to the public Rekor log. That's why the verify step passes --insecure-ignore-tlog=true. In CI the job signs keylessly instead, as the workflow below shows; "Signing, verifying and promoting by digest" (Advanced container security) explains both modes. Two registry:3 containers stand in for the build and production registries.

The pipeline script

ci.sh
#!/bin/sh
# Usage: ./ci.sh VERSION
# The pipeline one commit goes through. The CI job runs this same script; locally it uses
# cosign.key with COSIGN_PASSWORD, in GitHub Actions the sign step is keyless (see the workflow).
# Stops at the first failing stage. Nothing reaches the registry before test and scan pass.
set -eu
VERSION=$1
REPO=${BUILD_REPO:-localhost:5000/lab-build/app}
SIGN="--key cosign.key --signing-config lab-signing-config.json"
VERIFY="--key cosign.pub --insecure-ignore-tlog=true"
stage() { printf '\n== %s\n' "$*"; }
stage "1 build $REPO:$VERSION"
# the default (docker driver) builder: with the containerd image store, --load keeps the index
docker buildx build --builder default -q --provenance=mode=max --metadata-file build.json \
-t "$REPO:$VERSION" --load app
built=$(jq -r '."containerimage.digest"' build.json)
echo "built $built"
stage "2 test: start the image, call it, stop it"
docker run -d --name lab-ci-test -p 127.0.0.1:3900:3000 "$REPO:$VERSION" >/dev/null
trap 'docker rm -f lab-ci-test >/dev/null 2>&1' EXIT
for i in $(seq 1 30); do
[ "$(docker inspect -f '{{.State.Health.Status}}' lab-ci-test)" = healthy ] && break; sleep 1
done
curl -fsS http://127.0.0.1:3900/ | grep -q '"service":"payments"'
docker rm -f lab-ci-test >/dev/null
echo "test passed"
stage "3 scan: fail on HIGH or CRITICAL with a fix available"
trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$REPO:$VERSION"
echo "scan passed"
stage "4 push the image that was tested and scanned"
docker push -q "$REPO:$VERSION"
stage "5 resolve the digest"
digest=$(docker buildx imagetools inspect "$REPO:$VERSION" --format '{{.Manifest.Digest}}')
[ "$digest" = "$built" ] || { echo "registry has $digest, the build produced $built"; exit 1; }
echo "$digest" | tee "release-$VERSION.digest"
stage "6 attest: SBOM (signed) next to BuildKit's provenance"
syft scan -q "registry:$REPO@$digest" -o spdx-json=sbom.spdx.json
cosign attest --yes $SIGN --type spdxjson --predicate sbom.spdx.json "$REPO@$digest"
stage "7 sign the digest"
cosign sign --yes $SIGN "$REPO@$digest"
stage "8 verify before anyone may promote it"
cosign verify $VERIFY "$REPO@$digest" >/dev/null 2>&1
cosign verify-attestation $VERIFY --type spdxjson "$REPO@$digest" >/dev/null 2>&1
echo "VERIFIED $REPO@$digest"

Step 1 builds once with docker buildx build --load and --provenance=mode=max, and --metadata-file records the digest the build produced. The build names the default builder, which uses the docker driver. On Docker 29 with the containerd image store, --load from it keeps the image index with its provenance attestation. So the image in your local store is exactly what gets pushed. A docker-container builder loads a single manifest without the attestation ("Buildx builders, outputs, cache and Bake"). If you've left one selected with docker buildx use, step 5 would then fail on a digest mismatch. Step 2 starts that image, waits for its health check and calls it. A real project also runs its unit tests, either before the build or in a test stage; this step tests the artifact. Step 3 scans the same local image and fails on HIGH or CRITICAL findings that have a fixed version. Step 5 asks the registry for the digest behind the pushed tag. It refuses to continue unless that equals the build's. Steps 6 to 8 attach a Syft SBOM as a signed attestation, sign the digest and verify both.

Run 1: a vulnerable dependency

The app pins minimist 1.2.5, which has a known prototype-pollution vulnerability:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ grep -A1 '"node_modules/minimist"' app/package-lock.json
"node_modules/minimist": { "version": "1.2.5",
$ ./ci.sh 1.5.0-ci.1
== 1 build localhost:5000/lab-build/app:1.5.0-ci.1 sha256:b70eb289488eabb92cc1232007345b52ef7323ff0c2dc30b870860f7a26a061d built sha256:b70eb289488eabb92cc1232007345b52ef7323ff0c2dc30b870860f7a26a061d == 2 test: start the image, call it, stop it test passed == 3 scan: fail on HIGH or CRITICAL with a fix available ... Node.js (node-pkg) ================== Total: 1 (HIGH: 0, CRITICAL: 1) ┌─────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬────────────────────────────────────────────┐ │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │ ├─────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼────────────────────────────────────────────┤ │ minimist (package.json) │ CVE-2021-44906 │ CRITICAL │ fixed │ 1.2.5 │ 1.2.6, 0.2.4 │ minimist: prototype pollution │ │ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2021-44906 │ └─────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴────────────────────────────────────────────┘
$ curl -s http://localhost:5000/v2/_catalog; echo
{"repositories":[]}

Build and test passed. The scan found CVE-2021-44906, CRITICAL, with fixed versions available, and --exit-code 1 stopped the script. The build registry's catalog is empty: the image that failed the gate exists only on the build machine. --ignore-unfixed keeps the gate on findings you can act on today. Findings without a fix still belong in a report and a risk decision, not in a blocked merge. "Pinning, SBOMs, provenance and scanning" (Advanced container security) covers severity policy and VEX.

Run 2: the fix

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ diff app/package.json fix/package.json cp fix/package.json fix/package-lock.json app/
7c7 < "minimist": "1.2.5", --- > "minimist": "1.2.8",
$ ./ci.sh 1.5.0-ci.2
== 1 build localhost:5000/lab-build/app:1.5.0-ci.2 sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 built sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 == 2 test: start the image, call it, stop it test passed == 3 scan: fail on HIGH or CRITICAL with a fix available Report Summary ┌─────────────────────────────────────────────────────────┬──────────┬─────────────────┬─────────┐ │ Target │ Type │ Vulnerabilities │ Secrets │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ localhost:5000/lab-build/app:1.5.0-ci.2 (alpine 3.24.3) │ alpine │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/node_modules/minimist/package.json │ node-pkg │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/node_modules/ms/package.json │ node-pkg │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/package.json │ node-pkg │ 0 │ - │ └─────────────────────────────────────────────────────────┴──────────┴─────────────────┴─────────┘ ... scan passed == 4 push the image that was tested and scanned localhost:5000/lab-build/app:1.5.0-ci.2 == 5 resolve the digest sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 == 6 attest: SBOM (signed) next to BuildKit's provenance Using payload from: sbom.spdx.json Signing artifact... == 7 sign the digest Signing artifact... Pushing signature to: localhost:5000/lab-build/app == 8 verify before anyone may promote it VERIFIED localhost:5000/lab-build/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1

Every stage passed. The digest the build printed (built sha256:...) and the one the registry returned after the push are the same. The script wrote it to release-1.5.0-ci.2.digest, the record the rest of the release uses. Your digests will differ, since they hash content that includes build timestamps. The provenance attestation inside the pushed index records what the build used:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ docker buildx imagetools inspect localhost:5000/lab-build/app@$(cat release-1.5.0-ci.2.digest) \ --format '{{json .Provenance}}' | jq -r '.SLSA.buildDefinition.resolvedDependencies[].uri'
pkg:docker/node@26-alpine?digest=sha256:40cd011b55415bc894e45b02fdd5a5dfde056921b7d23a64c74f80497af48797&platform=linux%2Farm64

The base image by digest and platform. Because the signature covers the index digest, and the index lists the attestation manifest, the same signature covers the provenance. The SBOM is attached on its own with cosign attest, signed with the same key. Many policy tools look for a signed SBOM attestation.

Promotion

promote.sh
#!/bin/sh
# Usage: ./promote.sh SOURCE_REPO@DIGEST TARGET_REPO:TAG
# Promotion job: verify the signature and the SBOM attestation on the exact digest, copy the image by
# digest and its signature artifacts (tag sha256-<hex>), verify both again in the target, and only then
# point the release tag at it. Being in the build registry is not an approval; only a digest that
# verifies is promoted, and a failed copy never leaves the release tag on an unverified image.
set -eu
src=$1 target=$2
VERIFY="--key cosign.pub --insecure-ignore-tlog=true"
case $src in *@sha256:*) ;; *) echo "REFUSED $src: promote a digest, not a tag"; exit 1 ;; esac
digest=${src#*@}; srcrepo=${src%@*}; dstrepo=${target%:*}
verified() { # verified REPO@DIGEST: the signature and the SBOM attestation both verify
cosign verify $VERIFY "$1" >/dev/null 2>&1 &&
cosign verify-attestation $VERIFY --type spdxjson "$1" >/dev/null 2>&1
}
verified "$src" || { echo "REFUSED $src: no valid signature and SBOM attestation"; exit 1; }
# On registries without the OCI referrers API (registry:3 here) Cosign keeps the signature and the
# attestation under this tag. A registry with that API has no such tag; copy with `oras cp -r` there.
sigtag=sha256-${digest#sha256:}
docker buildx imagetools inspect "$srcrepo:$sigtag" >/dev/null 2>&1 ||
{ echo "REFUSED $src: no $sigtag tag to copy"; exit 1; }
docker buildx imagetools create --progress=quiet --tag "$dstrepo@$digest" "$src"
docker buildx imagetools create --progress=quiet --tag "$dstrepo:$sigtag" "$srcrepo:$sigtag"
verified "$dstrepo@$digest" || { echo "FAILED $dstrepo@$digest does not verify in the target"; exit 1; }
docker buildx imagetools create --progress=quiet --tag "$target" "$dstrepo@$digest"
echo "PROMOTED $dstrepo@$digest as $target"

The promotion job takes a digest, never a tag. It verifies the signature and the SBOM attestation. Then it copies the image registry-side by digest with docker buildx imagetools create, which moves no tag, and copies the signature artifacts too. You need that second copy. imagetools create copies the image index and what it references, but Cosign's signature and attestation live in a separate artifact. In this registry that artifact is tagged sha256-<hex of the digest>. The job verifies both again in the target, and only then points the release tag at the digest. If a copy or the second check fails, the release tag still names the last release. Build to staging, then staging to production:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ ./promote.sh localhost:5000/lab-build/app@$(cat release-1.5.0-ci.2.digest) localhost:5000/lab-staging/app:1.5.0
PROMOTED localhost:5000/lab-staging/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 as localhost:5000/lab-staging/app:1.5.0
$ ./promote.sh localhost:5000/lab-staging/app@$(cat release-1.5.0-ci.2.digest) localhost:5001/lab-prod/app:1.5.0 curl -s http://localhost:5001/v2/lab-prod/app/tags/list; echo
PROMOTED localhost:5001/lab-prod/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 as localhost:5001/lab-prod/app:1.5.0 {"name":"lab-prod/app","tags":["1.5.0","sha256-3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1"]}
$ docker run -d --name lab-ci-prod -p 127.0.0.1:3901:3000 \ localhost:5001/lab-prod/app@$(cat release-1.5.0-ci.2.digest) >/dev/null sleep 2; curl -s http://127.0.0.1:3901/
Unable to find image 'localhost:5001/lab-prod/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1' locally localhost:5001/lab-prod/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1: Pulling from lab-prod/app Digest: sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 Status: Downloaded newer image for localhost:5001/lab-prod/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 {"service":"payments","uptime":"2s"}

Production holds the release tag and the signature tag, and the container runs from the production registry by digest. Deploy by digest, not by tag, so a later push to the tag can't change what's running.

The sha256-<hex> tag is the fallback that Cosign 3 uses on registries without the OCI 1.1 referrers API, such as registry:3. On a registry that has the API, the signature and the attestation are stored as referrers of the digest, and no such tag exists. In an earlier test with zot 2.1, a registry with that API, Cosign 3.1.3 created no sha256- tag. So promote.sh stops at its tag check before it copies anything. For such registries, copy the image together with its referrers using oras cp -r. In the same test, ORAS 1.3.4 copied them between zot and registry:3 in both directions, and cosign verify passed on the copies. cosign copy is deprecated in Cosign 3 and didn't carry the new signature format.

Now the bypasses. A tag is refused outright, because it could point anywhere by the time the copy runs. A hotfix built on someone's machine and pushed straight into the build repository is refused because nothing signed it:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ ./promote.sh localhost:5000/lab-build/app:1.5.0-ci.2 localhost:5001/lab-prod/app:1.5.0
REFUSED localhost:5000/lab-build/app:1.5.0-ci.2: promote a digest, not a tag
$ docker build -q --build-arg HOTFIX=1 -t localhost:5000/lab-build/app:hotfix app docker push -q localhost:5000/lab-build/app:hotfix ./promote.sh localhost:5000/lab-build/app@$(docker buildx imagetools inspect localhost:5000/lab-build/app:hotfix --format '{{.Manifest.Digest}}') \ localhost:5001/lab-prod/app:1.5.0
sha256:b85df851a7b497f199bd1295d84b0d8f7156f31e56c82d16a0a9e52fb5d5b558 localhost:5000/lab-build/app:hotfix REFUSED localhost:5000/lab-build/app@sha256:b85df851a7b497f199bd1295d84b0d8f7156f31e56c82d16a0a9e52fb5d5b558: no valid signature and SBOM attestation
$ docker buildx imagetools inspect localhost:5001/lab-prod/app:1.5.0 --format '{{.Manifest.Digest}}'
sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1

The hotfix push succeeded; the registry accepts anything from someone with push rights. Promotion didn't, and production still points at the signed digest. Because promotion checks the signature, you can share the build registry without trusting it. An image that's there proves nothing.

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.9.2
$ docker rm -f lab-ci-prod docker rm -f -v lab-registry lab-prod-registry docker image rm localhost:5000/lab-build/app:1.5.0-ci.1 localhost:5000/lab-build/app:1.5.0-ci.2 \ localhost:5000/lab-build/app:hotfix localhost:5001/lab-prod/app@$(cat release-1.5.0-ci.2.digest) rm -f cosign.key cosign.pub lab-signing-config.json build.json sbom.spdx.json release-*.digest rm -rf ~/lab/bin ~/.cache/trivy ~/.sigstore
... Untagged: localhost:5001/lab-prod/app@sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1 Deleted: sha256:3b17976474471ef21108352a052ab79739a205fc3a9bd1bcd2edda290847b2b1

The same pipeline in GitHub Actions

The workflow below was linted, not run. The lab has no GitHub runner, and keyless signing needs a GitHub OIDC identity, so that part is only described here. It shows the build job and the production promotion. A staging promotion is the same job with the staging repository and an environment of its own.

.github/workflows/release.yml
name: release
on:
push:
branches: [main]
permissions: {}
env:
BUILD_REPO: ghcr.io/acme/payments-api/build
PROD_REPO: registry.example.com/prod/payments-api
# The exact identity a signature must carry: this workflow file on main.
SIGNER: https://github.com/acme/payments-api/.github/workflows/release.yml@refs/heads/main
ISSUER: https://token.actions.githubusercontent.com
jobs:
build:
runs-on: ubuntu-24.04
permissions:
contents: read
packages: write # push to ghcr.io
id-token: write # OIDC token for keyless signing
outputs:
digest: ${{ steps.digest.outputs.digest }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# Same Engine as the lab, with the containerd image store so --load keeps the index.
- uses: docker/setup-docker-action@2bf61fb9464cc67f0cbdeabed6aa0380accd1c70 # v5.5.0
with:
version: v29.9.2
daemon-config: '{"features": {"containerd-snapshotter": true}}'
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
driver: docker
- name: Install cosign, syft and trivy (pinned, checksum-verified)
run: BIN="$RUNNER_TEMP/bin" ./install-tools.sh cosign syft trivy && echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
- name: 1 build
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: app
load: true
provenance: mode=max
tags: ${{ env.BUILD_REPO }}:${{ github.sha }}
- name: 2 test
run: |
docker run -d --name test -p 127.0.0.1:3900:3000 "$BUILD_REPO:$GITHUB_SHA"
for _ in $(seq 1 30); do
[ "$(docker inspect -f '{{.State.Health.Status}}' test)" = healthy ] && break; sleep 1
done
curl -fsS http://127.0.0.1:3900/ | grep -q '"service":"payments"'
docker rm -f test
- name: 3 scan
run: trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$BUILD_REPO:$GITHUB_SHA"
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: 4 push and 5 resolve the digest
id: digest
env:
BUILT: ${{ steps.build.outputs.digest }}
run: |
docker push -q "$BUILD_REPO:$GITHUB_SHA"
d=$(docker buildx imagetools inspect "$BUILD_REPO:$GITHUB_SHA" --format '{{.Manifest.Digest}}')
[ "$d" = "$BUILT" ] || { echo "pushed $d, built $BUILT"; exit 1; }
echo "digest=$d" >> "$GITHUB_OUTPUT"
- name: 6 attest, 7 sign (keyless), 8 verify
env:
DIGEST: ${{ steps.digest.outputs.digest }}
run: |
IMAGE="$BUILD_REPO@$DIGEST"
syft scan -q "registry:$IMAGE" -o spdx-json=sbom.spdx.json
cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$IMAGE"
cosign sign --yes "$IMAGE"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$IMAGE" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$IMAGE" > /dev/null
promote:
needs: build
runs-on: ubuntu-24.04
# The production environment has required reviewers and allows only the main branch. Its own
# secrets (PROD_REGISTRY_USER/TOKEN) exist only for jobs that run in it, after approval.
environment: production
permissions:
contents: read
packages: read
env:
DIGEST: ${{ needs.build.outputs.digest }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Install cosign (pinned, checksum-verified)
run: BIN="$RUNNER_TEMP/bin" ./install-tools.sh cosign && echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: registry.example.com
username: ${{ secrets.PROD_REGISTRY_USER }}
password: ${{ secrets.PROD_REGISTRY_TOKEN }}
- name: Verify, then promote by digest
run: |
SRC="$BUILD_REPO@$DIGEST"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$SRC" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$SRC" > /dev/null
# Copy by digest, then the signature tag, verify in production, and only then tag.
# (A registry with the OCI referrers API has no sha256-<hex> tag: use `oras cp -r` there.)
sigtag="sha256-${DIGEST#sha256:}"
docker buildx imagetools create --tag "$PROD_REPO@$DIGEST" "$SRC"
docker buildx imagetools create --tag "$PROD_REPO:$sigtag" "$BUILD_REPO:$sigtag"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$PROD_REPO@$DIGEST" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$PROD_REPO@$DIGEST" > /dev/null
docker buildx imagetools create --tag "$PROD_REPO:${GITHUB_SHA::12}" "$PROD_REPO@$DIGEST"

Every third-party action is pinned to a full commit SHA, with the release in a comment for humans and for Dependabot, which updates both. A tag such as @v4 can be moved by whoever controls the repository, or by whoever steals their credentials. In March 2026, attackers force-pushed 76 of 77 tags of aquasecurity/trivy-action to commits that stole CI secrets (CVE-2026-33634). This workflow doesn't use a scanner action at all. It installs Cosign, Syft and Trivy with the same install-tools.sh as the lab. It checks them against the hashes in the repo, so CI and the lab run the same binaries.

docker/setup-docker-action gives the job Docker Engine 29.9.2 with the containerd image store. So load: true keeps the provenance index exactly as in the lab, and setup-buildx-action with the docker driver builds inside that engine. build-push-action builds once with load: true, and its digest output is compared with what the registry reports after docker push. The job grants itself only what it needs: packages: write to push to GHCR and id-token: write to request an OIDC token.

That token is what keyless signing uses. cosign sign --yes IMAGE@DIGEST with no key gets a token from GitHub that names the repository, workflow file and ref. Sigstore's Fulcio issues a short-lived certificate for that identity. The signature and certificate go into the Rekor transparency log. There's no long-lived signing key to leak. Your verify step must then pin the exact identity: --certificate-identity with the workflow file and ref, and the GitHub issuer. Don't use a regular expression such as ^https://github.com/acme/. It accepts a signature from any workflow on any branch of any repo under that prefix. That includes a pull-request workflow an attacker can edit. The promote job runs in a GitHub environment with required reviewers, and it checks that identity. Only then does it copy the digest and its signature to the production registry.

The reviewers protect production only if no other job can reach the production secrets. Store PROD_REGISTRY_USER and PROD_REGISTRY_TOKEN as secrets of the production environment, not of the repo. Then limit that environment's deployment branches to main. Any workflow can read a repository secret, on any branch that someone with write access pushes. Such a workflow could push to production without the promote job. If your registry supports it, replace the token with OIDC federation, so there's no static production credential at all.

A scan at build time describes the image on the day it was built. New CVEs get published against images that are already in production. So run a scheduled job that rescans the promoted digests and opens a rebuild when a fix appears. The signed SBOMs make that cheap: trivy sbom scans an SBOM file without pulling the image. Shared runners also fetch the Trivy vulnerability database on every job. Cache it between runs or mirror it, since the download is rate-limited.

Builders for untrusted code

A build runs the code it builds: every RUN line, every package install script. For pull requests from forks, treat the build as hostile.

Do not mount /var/run/docker.sock into a job, and do not run jobs on a self-hosted runner whose user is in the docker group. If you can reach the socket, you're root on the runner, read-only mount or not. A job with root on a shared runner can read other jobs' secrets and replace the images they push. "The Docker socket and daemon hardening" (Advanced container security) explains why and what to use instead. Docker-in-Docker with --privileged is no better: a privileged container is root on the host.

Buildx itself is a client and isolates nothing. With the docker driver the build runs inside dockerd. With the docker-container driver it runs in a BuildKit container that's privileged by default ("Buildx builders, outputs, cache and Bake"). Here's what does isolate. GitHub-hosted runners or other throwaway VMs are destroyed after each job. Rootless BuildKit (the moby/buildkit:rootless image) runs without root, but needs relaxed seccomp and AppArmor settings for its user namespaces. Remote builders reached with the remote or kubernetes Buildx drivers leave the runner holding no daemon at all. You can also run rootless Docker on the runner ("Rootless Docker" in Advanced container security). Kaniko was the usual answer for years. Google archived it in June 2025, and Chainguard keeps a community fork. Whatever builds untrusted code must also hold no registry login and no signing identity. GitHub withholds secrets from fork pull requests, but pull_request_target workflows that check out the pull request's code undo that protection.

Try this

Before you run the cleanup step, push a second unsigned image the way step 15 did. Give it a different --build-arg value and its own tag, then pass its digest to ./promote.sh. Predict which REFUSED line you'll get, then check it against step 15. Afterwards, rerun the prod-unchanged command and confirm the production tag still resolves to the digest in release-1.5.0-ci.2.digest.

Takeaway

Treat a digest as the release, and make every gate refer to it: one build, then scan, push, sign and promote that exact value. The registry happily took the unsigned hotfix push, so a registry that accepts an image isn't evidence of anything. Pipelines go wrong when promotion takes a tag or skips the verify step. That's the exact path promote.sh refused with promote a digest, not a tag.

Next: Advanced container security. The Docker in depth course ends here. The advanced course starts from the kernel side. You'll see what namespaces, cgroups and capabilities really enforce, and how you harden the image, the container and the daemon. Then it shows how escapes work and how they're stopped.

Quick check
01A workflow builds with load: true, scans the local image with Trivy, then runs build-push-action again with push: true, and signs steps.push.outputs.digest. What is wrong?
Incorrect — Cosign signs any manifest digest in a registry; the output is fine to use.
Correct — The second build is a new artifact. Build once, push that image, and check that the pushed digest equals the build's.
Incorrect — A signature is over a digest that already exists in a registry, so signing comes after the push.
Incorrect — Trivy scans local engine images, tarballs and registry references.
02A developer pushes a hotfix image directly into the build repository, where the CI job also pushes. The promotion job is given the hotfix's digest. What should stop it reaching production?
Incorrect — A registry accepts any push from someone with push rights; the lab's hotfix push succeeded.
Incorrect — A clean scan says nothing about who built the image or whether it went through the pipeline.
Incorrect — A digest identifies content exactly, including content nobody approved.
Correct — Nothing signed the hotfix, so verification fails and the copy never runs.
03Production verification uses --certificate-identity-regexp '^https://github.com/acme/api/' with the GitHub OIDC issuer. Why is that weaker than --certificate-identity https://github.com/acme/api/.github/workflows/release.yml@refs/heads/main?
Correct — Pin the exact workflow file and ref that is allowed to sign releases.
Incorrect — Both forms check the log; they differ in which identities they accept.
Incorrect — They are supported, which is why the loose form is a real risk.
Incorrect — A signature proves who signed which digest, not what the image contains.

Related