Signing, verifying and promoting by digest

Cosign v3 key-based and keyless signatures, verify-before-run, SBOM attestations and promotion by digest.

A production host deploys with docker pull registry.example.com/app:1.4 && docker run .... Nothing in that command can tell you which image app:1.4 is. It might be the one the pipeline built, tested and scanned. Or somebody might have pushed it with a leaked registry token an hour ago. The registry accepted both pushes without asking. A registry stores images; it doesn't approve them. Approval has to be something a deploy can check. One such check is a signature over the image digest. It's made after the image passed its tests, by a key or identity the deploy trusts.

You'll run the whole release order end to end with Cosign v3 and two local registries. You'll prove that a verify-before-run script refuses an unsigned image and a modified one. Then you'll promote the signed digest and verify real keyless signatures from public projects. Use the main lab VM. The lesson files go in ~/lab/signing: app/Dockerfile, install-tools.sh (from "Pinning, SBOMs, provenance and scanning") and verify-run.sh. The lab needs internet access to GitHub, gcr.io and Sigstore's public services.

The release order

Tests and scans run on the candidate before anything leaves the build machine. So a failing build never reaches a registry where someone could deploy it. After the push, the registry holds the image under the digest it'll be known by everywhere. Every later step uses that digest. Attestations and the signature are bound to it, verification checks it and promotion copies it. If you sign before the push, or sign a tag, you sign whatever the name resolves to at that moment. That isn't necessarily what was tested.

Use separate repositories for separate trust levels. The build registry gets every CI build and keeps it for a short time. Staging deploys from it by digest. Production pulls only from a registry (or repository) that the promotion job alone can write to. Every production host checks before it runs. In the lab, localhost:5000 is the build registry and localhost:5001 is production. "Docker in CI/CD: build to promote" in Docker in depth puts this order into a CI workflow.

Verify, then run

Docker Engine runs any image you ask it to run. It has no step that checks a signature before docker run. So on plain Docker hosts, verification is a script in front of the run. This one resolves the reference to a digest once, verifies a Cosign signature on that digest and runs exactly that digest:

#!/bin/sh
# Usage: ./verify-run.sh IMAGE[:TAG|@DIGEST] [COMMAND...]
# Resolves IMAGE to one digest, verifies a Cosign signature on that digest, and only then runs
# that digest. Exits 1 without running anything when verification fails.
# COSIGN_VERIFY is the trust policy, and there is no default: an unset policy is an error, so a copy
# of this script can never fall back to a weaker one. Examples:
#   keyless:  COSIGN_VERIFY="--certificate-identity=URI --certificate-oidc-issuer=URL"
#   key:      COSIGN_VERIFY="--key /etc/cosign/release.pub"   (an absolute path, never the working directory)
#   this lab: COSIGN_VERIFY="--key $HOME/lab/signing/cosign.pub --insecure-ignore-tlog=true"
set -eu
: "${COSIGN_VERIFY:?set COSIGN_VERIFY to the trust policy (see the comments in this script)}"
ref=$1; shift
name=${ref%%@*}
case ${name##*/} in *:*) name=${name%:*} ;; esac
digest=$(docker buildx imagetools inspect "$ref" --format '{{.Manifest.Digest}}')
image=$name@$digest
# COSIGN_VERIFY is a list of flags, so it is expanded unquoted on purpose.
if ! err=$(cosign verify $COSIGN_VERIFY "$image" 2>&1 >/dev/null); then
  echo "REFUSED  $image"
  printf '%s\n' "$err" | grep -m1 '^Error' | sed 's/^/         /'
  exit 1
fi
echo "VERIFIED $image"
exec docker run --rm "$image" "$@"

Running the digest instead of the name closes the gap between check and use. A tag can move between cosign verify and docker run. A digest can't, and Docker checks pulled content against it. COSIGN_VERIFY is the trust policy, and the script has no default. With the variable unset it stops with an error, so a copy on another host can never fall back to a weaker policy.

For keyless signatures, the policy names a certificate identity and issuer. For a key, it names the public key by an absolute path. That way a cosign.pub in whatever directory the script runs from is never trusted by accident. The lab sets a policy for its own key in the next section. That policy skips the transparency log, because the lab signs without one.

Tools, registries and a key pair

Install Cosign, Syft and Trivy with the same checksum-checked script you used in the previous lesson:

If GitHub resets the connection during a download, curl prints a curl: (35) line. Then --retry in the script tries again. A file that still came in damaged would fail its checksum line.

Start the build registry on port 5000 and the production registry on port 5001. Both are plain registry:3 containers. What makes one of them production is who may push to it.

A key pair is the simplest trust anchor. Whoever holds the private key can sign. If you have the public key, you can verify. The export COSIGN_VERIFY=... line in the next command sets the trust policy verify-run.sh uses for the rest of the lesson. Run it again in any new shell, together with the PATH and COSIGN_PASSWORD exports. cosign generate-key-pair encrypts the private key with the password in COSIGN_PASSWORD, here an obvious test value. In CI the password comes from the secret store. Or the key lives in a KMS and never touches disk (--key awskms://..., gcpkms://, azurekms://, hashivault://).

Cosign v3 takes the Sigstore services it uses (Fulcio, Rekor, a timestamp authority) from a signing config. The default comes from Sigstore's TUF repository. It uploads even key-based signatures to the public Rekor transparency log. lab-signing-config.json lists no services, so the lab publishes nothing about throwaway keys and a localhost registry. The price is that verification must skip the log (--insecure-ignore-tlog=true). And nothing independent records when a signature was made. In production, sign keyless (below) or keep a log, public or a private Rekor. The older --tlog-upload=false is deprecated, and v3 refuses it while a signing config is in use.

Build, test and scan the candidate

# alpine:3.24, pinned to the index digest the tag pointed at when this lab was written
FROM alpine:3.24@sha256:be2656061b536c1883c4034790e21d0a09c5aaaf51968b89368a3c0371ea2c9a
ARG BUILD=1
RUN echo "lab-app build $BUILD" > /build.txt
USER 10001
CMD ["cat", "/build.txt"]

lab-app:candidate exists only in your local image store. The test compares the output with what this build must print. The scan applies the gate from the previous lesson. Trivy downloads its database on this first run. The Alpine base has no fixable HIGH or CRITICAL findings today, so the gate passes. On another day it may not, and then the release stops here.

Push and resolve the digest

The 1.0: digest: line of the push output is the digest the registry stored for the tag (the index digest on Docker 29). sed saves it in app.digest. Do not ask the registry what 1.0 points to after the push. Another job could push the same tag in between, and you'd sign its image. With docker buildx build --push the digest comes from --metadata-file, and docker/build-push-action returns it as the digest output.

Attest and sign the digest

An attestation is a signed statement about the image. It's an in-toto statement whose subject is the digest and whose predicate is a document, here the SBOM. BuildKit's attestations from the previous lesson sit inside the image index and aren't signed. cosign attest signs one with your key, so a verifier knows who vouches for it. Generate the SBOM from the pushed digest, attest it, then sign:

Cosign v3 stores each signature and attestation as a Sigstore bundle, an OCI artifact that refers to the image digest. A registry with the OCI 1.1 referrers API returns such artifacts when asked for the referrers of a digest. registry:3 lacks that API. So Cosign used the fallback the OCI specification defines: a tag named after the digest, sha256-<hex>. Its index lists both bundles, the signature (cosign/sign/v1) and the SBOM attestation (spdx.dev/Document). The image index and its digest are unchanged.

Verify

cosign verify checked each signature against cosign.pub, and checked that the signed claims name this digest. It lists every bundle this key signed for the digest, so the attestation appears next to the signature; type tells them apart. The summary says the log was "verified offline", although the warning above says it was skipped. Trust the warning. verify-attestation returns the signed statement, which decodes to the SBOM for this digest. You'd build a deploy policy on this command when it requires an attestation of a given type (an SBOM, a scan result, a VEX document).

Now the script. It'll run the signed image:

Someone builds a "hotfix" and pushes it straight to the production registry, bypassing the pipeline. The push works, because the registry checks credentials, not provenance. The script doesn't run it:

Here's a second case. Someone rebuilds the image, pushes it over 1.0 in the build registry and signs it with a key of their own. So the tag now points at a signed image:

The tag resolved to the new digest, whose only signature was made with other.key. Cosign v3 says "no matching attestations" for signature bundles too. The important part is "Found: 0, Expected 1": no signature from the trusted key. A check for "any valid signature" would have passed. That's why a signature means nothing without the key or identity you require. The tested image is still reachable by the recorded digest, and that still verifies:

Docker had no local image under that digest reference, so it pulled one. Every layer was already in your local store, and Docker checked the content against the digest.

Promote by digest

Promotion copies the verified digest from the build registry to production with imagetools create, as in "Tags, digests and promotion" in Docker in depth. Then verify in production:

The image arrived with the same digest, but the signature didn't come with it. imagetools create copies the index and everything it references. The signature bundles reference the image, not the other way round, so nothing in the index leads to them. Copy them as well. With the tag fallback, they're one more tag to copy:

The signature covers the digest, not a registry name. So it verifies wherever the digest and its signature artifacts are. Registries with the referrers API have no fallback tag to copy. There, use a tool that copies referrers with the image, such as oras copy -r or regctl image copy --referrers. (cosign copy is deprecated in v3, and its help points to oras copy -r.) The other common pattern is to sign again after promotion. You'd use a production key that only the promotion job holds. Then the second signature means "approved for production" rather than "built by CI".

Keyless signing

Keys have to be stored, rotated and revoked, and that's work you'd rather skip. Keyless signing swaps the key for an identity. The CI job requests an OpenID Connect (OIDC) token from its platform. That's a signed statement such as "workflow release.yaml in repository org/app, running for tag v1.4". Cosign generates a throwaway key pair. Fulcio, Sigstore's certificate authority, issues a ten-minute certificate binding that public key to the identity. Cosign signs the digest, and Rekor records the signature with a timestamp in a public append-only log. Then the private key is discarded.

A verifier checks three things. The certificate must chain to Fulcio's root (distributed through TUF). The signature must have been logged while the certificate was valid. And identity and issuer must match what it expects. Public Fulcio accepts tokens only from configured issuers (GitHub Actions, GitLab.com, Google and others). A self-hosted GitLab needs a private Sigstore deployment.

Keyless signing needs a real OIDC identity. So the lab checks signatures that public projects made. Google signs its distroless images with a Google service account identity. The distroless README names it:

You verify by digest again. Subject and issuer come from the Fulcio certificate. The Rekor entry shows when the signature was logged. With a different expected identity, the same signature is rejected, and the error names the identity it found. Pin the identity exactly. A loose --certificate-identity-regexp '.*' accepts a signature from anyone the issuer will vouch for. For GitHub Actions, that means any workflow in any repository. If you need a pattern, anchor it (^https://github\.com/org/app/\.github/workflows/release\.yaml@refs/tags/v).

Signatures from GitHub Actions carry the workflow file and Git ref as identity. Trivy's release checksums are signed that way. That also completes the install from the previous lesson, since you can verify the checksum file that the hashes in install-tools.sh were taken from:

The curl: (35) lines are connection resets that --retry got past, and Verified OK confirms the checksum file came from Trivy's release workflow at tag v0.76.1. The identity says which workflow on which ref signed. It doesn't say the code was safe. In the March 2026 Trivy compromise, the attacker pushed a commit that altered the release workflow and tagged it v0.69.4. The project's own release pipeline then built and published the malicious release from that tag. Protected tags and branches, and review of workflow changes, decide whether you can trust a ref. In CI, signing is cosign sign --yes IMAGE@DIGEST with no key, plus permission to request an OIDC token (id-token: write on GitHub Actions). "Docker in CI/CD: build to promote" in Docker in depth shows the workflow.

Where verification can be enforced

Docker Engine itself won't ever enforce a signature. Docker Content Trust, the Notary v1 mechanism behind DOCKER_CONTENT_TRUST=1 and docker trust, was removed from the Docker CLI in 29.0:

The variable is silently ignored, and the pull just works. On Docker hosts, you have three enforcement points. There's a wrapper like verify-run.sh, and a deploy job that verifies before it calls Docker. And there's a production registry that only a verifying promotion job can write to. Engine authorization plugins can veto API calls, but none that ships with Docker checks Sigstore signatures. Kubernetes has the step Docker lacks. An admission controller sees every pod before it's created. Kyverno or Sigstore's policy-controller reject pods whose images lack a valid signature or attestation from the required identity.

Notation, from the CNCF Notary Project, is the other current signing tool. It signs digests with X.509 certificates from your own PKI (often a cloud key vault). It stores signatures as OCI referrers and verifies against a trust policy file. Cosign suits you if you want keyless CI identities and a public log. Notation suits you if you already run certificate authorities. Both sign digests, both need verification wired into the deploy path, and Kubernetes admission controllers support both.

Clean up

Remove the registries and the images, including the digest references the runs made. Then remove the keys, the files you made, the tools and their caches:

Try this

Before the clean-up step, on the main lab VM in ~/lab/signing, run the script once with the trust policy removed: env -u COSIGN_VERIFY ./verify-run.sh localhost:5000/lab-build/app:1.0. Predict whether it resolves the tag or calls cosign at all, then read the : "${COSIGN_VERIFY:?...}" line in the listing and compare. Then, with the policy set again, run it on the production copy by digest, localhost:5001/lab-prod/app@$(cat app.digest). Predict whether it verifies now that the signature tag was copied, and compare with the promote-sig output above.

Remember

Check the digest you're about to run against the one key or identity you trust. Then run that same digest. The rebuilt image pushed over 1.0 carried a perfectly valid signature, and only the "Found: 0, Expected 1" check against the trusted key stopped it. The usual review miss is a loose --certificate-identity-regexp '.*' or an "any signature" check, which passes for whoever managed to sign.

Related