Pinning, SBOMs, provenance and scanning

Digest pins and the bots that move them, BuildKit attestations, Syft, Trivy and Grype gates, and VEX.

An advisory lands on a Tuesday: the Werkzeug debugger lets an attacker run code (CVE-2024-34069), fixed in Werkzeug 3.0.3. Security asks two questions. Which of the images you run contain Werkzeug older than 3.0.3? And in which of them can anyone reach the debugger? If you store a software bill of materials (SBOM) for every image digest, the first question is a search through files you already have. Without SBOMs you'd have to pull and unpack every image in every registry. The second question no scanner can answer for you.

You'll build a Flask image that's out of date on purpose, with BuildKit's SBOM and provenance attestations. Then you'll generate SBOMs with Syft, scan with Trivy and Grype, gate on the result and record a "not affected" decision as VEX. Signing and promotion come next, in "Signing, verifying and promoting by digest". Use the main lab VM. The lesson files go in ~/lab/provenance: app/ (Dockerfile, requirements.txt, app.py), install-tools.sh and vex.json. The lab needs internet access to GitHub, PyPI and the scanners' databases.

Install the tools

Syft, Trivy and Grype are static binaries, so you can run them straight from a folder. install-tools.sh downloads pinned releases from GitHub and checks each one against a SHA-256 value committed in the script itself. Then it installs them into ~/lab/bin, which the cleanup deletes. Nothing is installed system-wide.

#!/bin/sh
# Usage: ./install-tools.sh TOOL...   (TOOL: cosign syft trivy grype)
# Downloads pinned release binaries from GitHub, checks each against the SHA-256 committed below,
# and installs them into $BIN (default ~/lab/bin). Nothing system-wide.
# The hashes live in this file, in your repository: a release asset replaced on GitHub after you
# pinned it fails the check. Bumping a version means updating its two hashes in a reviewed change.
set -eu
BIN=${BIN:-$HOME/lab/bin}
COSIGN=3.1.4 SYFT=1.55.0 TRIVY=0.76.1 GRYPE=0.121.0
case $(uname -m) in
  aarch64|arm64) A=arm64 T=ARM64 ;;
  x86_64)        A=amd64 T=64bit ;;
  *) echo "unsupported CPU $(uname -m)" >&2; exit 1 ;;
esac
sha() {  # sha ASSET: the pinned SHA-256 of a release asset
  case $1 in
    cosign-linux-arm64)                echo ade83b7aac62e131cacfe96d958948b2150cc9efec46a570a2649b8e27dedad2 ;;
    cosign-linux-amd64)                echo 424e2c97d0b45fd7a6b608a87f090f35633f4c23ece5f092d48bba11aaf4bbfa ;;
    syft_1.55.0_linux_arm64.tar.gz)    echo 929bd163db2d7cd2255d3bcb8967208079813d7b46b0c952f7b794d8e2c7d304 ;;
    syft_1.55.0_linux_amd64.tar.gz)    echo bb65249542db5fedeeb4d3f9b018b75e7264dbfbbb10486f044581a92f90490d ;;
    trivy_0.76.1_Linux-ARM64.tar.gz)   echo e174dff2a6dba5e3ae41f8d0537a6324405bd08132497340304c6b20e0f16a1f ;;
    trivy_0.76.1_Linux-64bit.tar.gz)   echo c7cbe3137fb8a48a17574bede3dd6beb16742d1e9772929940fc90082c39d9c0 ;;
    grype_0.121.0_linux_arm64.tar.gz)  echo 86f570f75ca947ea3dd6486309232bcff6d78e44ba081ae75ab14eab72be3dae ;;
    grype_0.121.0_linux_amd64.tar.gz)  echo 64e07133d3486115de647a66581461dccf5c732748a0a3119077395565f15519 ;;
    *) echo "no pinned hash for $1" >&2; return 1 ;;
  esac
}
GH=https://github.com
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
mkdir -p "$BIN"
 
fetch() {  # fetch URL_DIR ASSET: download the asset and check it against the pinned hash
  want=$(sha "$2")
  curl -fsSL --retry 10 --retry-delay 5 --retry-all-errors -o "$tmp/$2" "$1/$2"
  (cd "$tmp" && echo "$want  $2" | sha256sum -c -)
}
 
for t in "$@"; do
  case $t in
    cosign) u=$GH/sigstore/cosign/releases/download/v$COSIGN
            fetch "$u" "cosign-linux-$A"
            install -m 0755 "$tmp/cosign-linux-$A" "$BIN/cosign" ;;
    syft)   u=$GH/anchore/syft/releases/download/v$SYFT
            fetch "$u" "syft_${SYFT}_linux_$A.tar.gz"
            tar -xzf "$tmp/syft_${SYFT}_linux_$A.tar.gz" -C "$BIN" syft ;;
    trivy)  u=$GH/aquasecurity/trivy/releases/download/v$TRIVY
            fetch "$u" "trivy_${TRIVY}_Linux-$T.tar.gz"
            tar -xzf "$tmp/trivy_${TRIVY}_Linux-$T.tar.gz" -C "$BIN" trivy ;;
    grype)  u=$GH/anchore/grype/releases/download/v$GRYPE
            fetch "$u" "grype_${GRYPE}_linux_$A.tar.gz"
            tar -xzf "$tmp/grype_${GRYPE}_linux_$A.tar.gz" -C "$BIN" grype ;;
    *) echo "unknown tool $t" >&2; exit 1 ;;
  esac
done

Each OK line is sha256sum -c accepting one download. The curl: (35) lines are connection resets from GitHub that the script's --retry got past. The hashes live in the script, in your repository. So a release asset that's replaced on GitHub after you pinned it fails the check. A checksum file downloaded from the same release can't catch that, since whoever can publish the binary can publish a matching checksum file. The hashes still have to be right when you first pin them. Take them from a release whose signature you verified (the next lesson verifies the signature on Trivy's checksum list). Bump a version by changing its hashes in a reviewed commit, as "Docker in CI/CD: build to promote" (Docker in depth) does. Run the export again in every new shell. Each tool also ships as a container image. If you use one, pin it by tag@digest, as "Slimming images and layer hygiene" does for dive. Trivy's own images were part of the incident below, and only references by digest were safe.

A pinning policy

In March 2026 an attacker with stolen credentials force-pushed 76 of the 77 version tags of the aquasecurity/trivy-action GitHub Action to credential-stealing code. The attacker also published a malicious Trivy release (CVE-2026-33634). Workflows that used the action by version tag ran the stealer. Trivy images referenced by digest weren't affected, and neither were workflows pinned to a recent commit SHA. Pins to commits older than April 2025 were still exposed, because that code fetched a helper action by tag. A pin covers what it names, not what the pinned code pulls in. You can pin every input to a build:

The lab Dockerfile pins its base by index digest. Its requirements file pins Flask and Werkzeug to 2.2.2, from 2022, so the scanners have something to find:

# python:3.15-slim, pinned to the index digest the tag pointed at when this lab was written
FROM python:3.15-slim@sha256:533b0b2f9b5d46fd097091cf826f27902f1d6c3bca714b6dc74551356aa97a51
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
USER 10001
CMD ["python", "app.py"]

flask==2.2.2
werkzeug==2.2.2

A pin nobody moves turns into a frozen flaw. It misses every Debian security fix after the day it was written. Give pins to a bot. Renovate and Dependabot notice when a tag points at a new digest, or a newer tag exists. They open a pull request that changes the pin, and the pipeline rebuilds, tests and scans it like any other change. Renovate's config:best-practices preset pins and updates digests for Docker images and GitHub Actions:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:best-practices"],
  "packageRules": [
    {
      "matchDatasources": ["docker"],
      "matchUpdateTypes": ["digest"],
      "automerge": true
    }
  ]
}

The package rule merges digest-only updates (same tag, rebuilt image) once the checks pass. New tags wait for a human. Dependabot does the same from .github/dependabot.yml. It updates the digest of a pinned FROM line together with its tag:

# version 2 is the dependabot.yml schema; do not confuse it with the obsolete Compose key
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/app"
    schedule:
      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/app"
    schedule:
      interval: "weekly"
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

Either way, every pin has a bot that moves it. Every move goes through build, test and scan, and nobody edits a digest by hand.

Attestations from the build

BuildKit can attach two attestations to an image: an SBOM (--sbom=true) and SLSA provenance, a record of how the image was built (--provenance). Both are in-toto statements, JSON documents that name the image by digest. They're stored in the attestation manifest that "Image anatomy: index, manifest, config and layers" in Docker in depth took apart. Provenance in min mode is on by default; the SBOM isn't. The default docker driver can produce them because fresh Docker 29 installs use the containerd image store. A host still on the legacy store refuses with "Attestation is not supported for the docker driver". Use a docker-container builder there ("Buildx builders, outputs, cache and Bake" in Docker in depth).

Start a local registry and build with both attestations, provenance in max mode. --metadata-file writes the build result, including the digest, to a JSON file:

The resolve image config and docker-image:// steps fetch docker/buildkit-syft-scanner, the SBOM generator. It's an image with Syft inside that BuildKit pulls from Docker Hub. The generating sbom step near the end runs it against the final stage. Earlier stages of a multi-stage build aren't scanned unless the Dockerfile declares ARG BUILDKIT_SBOM_SCAN_STAGE=true. (BUILDKIT_SBOM_SCAN_CONTEXT does the same for the build context.) containerimage.digest is the index digest, the value you sign, deploy and promote. app.digest keeps it for the rest of the lesson. It changes on every build because the config records the build time.

The attestation manifest now has two layers, one for each, and each is labelled with its predicate type:

The SPDX document is 2 MB because Syft lists every file it found as well as every package. The provenance is about 10 kB. Read the SBOM through imagetools, which unwraps the in-toto statement:

The creators name the generator, Syft 1.51.0 bundled with BuildKit 0.34.0. Keep that with the SBOM, because generator versions differ in what they list. The SBOM covers the Python packages pip installed and the Debian packages of the base image. For an image with more than one platform, select one platform with --format '{{ json (index .SBOM "linux/amd64").SPDX }}'.

The provenance answers "what was this built from":

resolvedDependencies lists the base image with the digest BuildKit used. It also lists the scanner image, whose digest is in the full record. That works even for an unpinned FROM, so provenance tells you which base a running image came from. The last line, a build step, exists only in max mode. Build again with the default min mode (-q prints only the new digest) and compare:

min holds the materials, the frontend, the names of the local contexts and timestamps, about 1 kB. max adds every build step with its command and environment, the Dockerfile text and the layers each step produced. That detail makes it useful in an incident. It's also why max publishes build-argument values next to the image ("Build-time secrets and how images leak them" shows the leak). max publishes the Dockerfile and every command with the image. So it's safe to use only once no credential appears in a build argument or anywhere on a RUN command line, such as a token in a pip install --index-url URL. Internal hostnames and registry paths in those commands ship too.

SBOMs from Syft

Syft is the standalone form of the generator BuildKit used. You need it for images you didn't build, such as vendor images and old releases, and for SBOM formats a customer or tool asks for. It reads the image from the registry and writes several formats in one pass:

SPDX (Linux Foundation) and CycloneDX (OWASP) describe the same packages in different schemas. Both identify a package by its package URL (purl), pkg:pypi/werkzeug@2.2.2, which is what scanners match against advisories. Trivy and Grype read both. Besides registry: (plain HTTP here, as Docker allows for localhost registries), Syft reads the local Docker store, docker save archives and OCI layouts.

Syft 1.55.0 lists 103 packages; BuildKit's SBOM of the same image listed 122. The difference:

pip ships its own SBOM, pip/_vendor/bom.cdx.json, describing the libraries it bundles. The Syft inside BuildKit read it and added those packages; standalone Syft 1.55.0 with default settings didn't. Neither is wrong. But a scan of an SBOM can only find what the SBOM lists, and the scans below show you what that costs. Syft also scans source trees:

A directory scan sees only what the manifests list: the two packages named directly, no Jinja2 or Click (pulled in by Flask), no Debian packages, no Python. It's useful for checking a repository before anything is built, not as a replacement for the image SBOM.

Scanning with Trivy and Grype

A scanner matches packages against a database of known flaws. Trivy downloads its database on the first run (into ~/.cache/trivy, about 1.4 GB unpacked on this VM). Scan the image by digest, HIGH and CRITICAL only, and count the findings by fix status:

None of the 44 Debian findings has a fix. affected means Debian hasn't released a patched package, and fix_deferred means Debian decided not to fix it in this release. Rebuilding can't clear them. A gate that counts them fails every build until Debian acts or you change the base. The Python findings all have fixed versions. Gate on what the team can fix, and keep the rest in view. Store the full report as a non-blocking artifact of the job, and review the unfixed HIGH and CRITICAL findings on a schedule with an owner. An unfixed CRITICAL that's being exploited is a reason to change the base image, not to wait for Debian.

--ignore-unfixed drops findings without a fix. --exit-code 1 makes Trivy exit 1 when anything remains, which fails the pipeline step. The first three are in packages the Dockerfile installs. The last four have no path: Trivy found them in pip's bundled SBOM, so they're libraries inside pip. Now scan the SBOM that Syft wrote instead of the image:

Three findings, not seven, because Syft's SBOM doesn't list pip's bundled libraries. Scanning stored SBOMs is fast and needs no registry access. That suits a nightly rescan of every release you've shipped (the image doesn't change; the advisories do). But it can only see what the generator saw. Grype, Anchore's scanner, reads the same inputs. Its database is larger (about 3 GB unpacked) and the first download takes a few minutes:

Grype reports GitHub advisory IDs (GHSA-2g68-c3qc-8985 is CVE-2024-34069). --only-fixed is its --ignore-unfixed but keeps every severity. So Medium and Low rows appear, all for Flask and Werkzeug in this run. On the earlier Python 3.14 image, Grype also listed the interpreter itself. --fail-on high makes Grype exit 2 when a High or Critical finding remains. Its High findings are the same three on the image and the SBOM. Grype catalogs images with Syft and misses pip's bundled libraries too. EPSS is the Exploit Prediction Scoring System estimate that a vulnerability will be exploited in the next 30 days. Use one scanner for the gate; two in one pipeline mostly produce two lists to reconcile. In CI, cache the scanner database between jobs or point the scanner at a mirror of it. Don't download gigabytes on every run and run into rate limits.

Known is not exploitable

Every finding above is a known flaw: a package version matches an advisory. Whether you can exploit it in this image is a different question. CVE-2024-34069 needs the Werkzeug debugger, which this app never enables. The urllib3, msgpack and setuptools findings are in pip's bundled libraries, which run only when someone runs pip inside the container. Nothing in production should. A multi-stage build that leaves pip out of the final image removes the findings together with the code. The Flask finding about session cookies only matters behind a caching proxy. EPSS adds the outside view of whether anyone is exploiting a vulnerability at all. None of that removes the need to patch. It decides what must block a release today and what can ride on next week's base-image bump.

VEX (Vulnerability Exploitability eXchange) records that decision in a machine-readable file. It holds the product, the vulnerability, a status (not_affected, affected, fixed, under_investigation) and, for not_affected, a justification from a fixed list. Scanners read it and stop reporting the finding. So you write the decision once, review it like code, and it stays with the artifact instead of living in a ticket. This OpenVEX document covers the debugger CVE:

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://example.test/vex/lab-app/2026-10-07",
  "author": "lab-app maintainers",
  "timestamp": "2026-10-07T00:00:00Z",
  "version": 1,
  "statements": [
    {
      "vulnerability": { "name": "CVE-2024-34069" },
      "products": [
        {
          "@id": "pkg:oci/lab-app?repository_url=localhost%3A5000%2Flab-app",
          "subcomponents": [ { "@id": "pkg:pypi/werkzeug@2.2.2" } ]
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path",
      "impact_statement": "The Werkzeug debugger is never enabled; app.py does not pass debug=True."
    }
  ]
}

The gate still fails, on the six findings that remain, and --show-suppressed lists what the VEX statement removed and why. Scope a statement as narrowly as the facts allow. The justification is a fact about this app. So the product is this image (pkg:oci/lab-app with its repository) and Werkzeug 2.2.2 is a subcomponent of it. VEX files get shared, published centrally and attached to base images. A statement that named only pkg:pypi/werkzeug@2.2.2 would hide this RCE in every image with that version, including a service that does run the debugger. The package version in the subcomponent also makes the next Werkzeug upgrade end the statement instead of hiding a new problem. Trivy also reads CycloneDX VEX and CSAF. The next lesson attaches documents like this one to the image as signed attestations.

Clean up

Remove the registry, the images, the generated files and the tools with their databases (about 4.5 GB):

Try this

Before you run the cleanup step, stay in ~/lab/provenance on the main lab VM with the tools still on your PATH. Run the trivy-sbom gate again, but point trivy sbom at sbom.spdx.json instead of sbom.cdx.json. Both came from the same Syft pass. Predict whether you'll get the same three findings, then compare with the CycloneDX result and with the seven from the image scan.

Bottom line

Generate and store an SBOM for every image digest, gate on fixable HIGH and CRITICAL findings, and record each "not affected" call as a narrowly scoped VEX statement. The trap is trusting an SBOM scan as complete. Here the image scan found seven gated findings, while the scan of Syft's SBOM found three, because that SBOM left out pip's bundled libraries. Keep the generator version with each SBOM, and remember that a VEX statement scoped to one image still left the gate failing on the six real findings.

Related