Signing & attesting the artifact

cosign signatures, SBOMs, and SLSA provenance in the pipeline.

Your Jenkins pipeline builds a container image, pushes it to a registry, and moves on. That image is now an unmarked box on a loading dock. Nobody who finds it later can tell whether your pipeline made it or whether someone swapped in a look-alike an hour after the push. The tag says checkout:1.4.2, but a tag is a sticky label, and sticky labels peel off and get stuck on other boxes. This lesson gives the image a history you can check: a signature that ties it to your pipeline, a list of everything inside it, and a signed record of how it was built. Then you put a guard on the deploy stage, so an image that fails those checks never reaches production.

An unsigned image is a package with no return address

A signature is a wax seal on a letter. Broken seal, and you know someone got in. Unbroken seal, stamped with a crest you recognize, and you keep reading. cosign (the tool that signs and verifies container images, part of the Sigstore project) does that job for an image. It ties the image's exact contents to the identity that signed them. Those contents are named by a SHA-256 digest. SHA-256 is a hashing algorithm, and the digest it produces is a fingerprint: change one byte anywhere in the image and the fingerprint changes. So when an attacker rebuilds the image with a backdoor in it and pushes it under the same tag, the digest is different, the old signature no longer matches, verification fails, and your deploy stage refuses to ship it. With no signature there is nothing to check at all, and the swapped image sails straight through.

This is why you sign the digest and never the tag. A tag like 1.4.2 is a name badge, and you can move a name badge onto somebody else whenever you feel like it. The digest sha256:9f2a... names one specific pile of bytes and nothing else. Sign that, and you have pinned down exactly what you are vouching for. Every step in this lesson points at that one digest, and that is what holds the chain together.

Keyless signing: an ID badge, not a house key

Keyless signing takes away the chore everybody dreads, which is minding a private key. A private key is a house key. You hide it, you rotate it, and you hope it never leaks. Keyless works more like showing your ID badge at a reception desk and walking off with a visitor pass that expires in an hour. Your pipeline proves who it is with an OIDC (OpenID Connect) token, a short-lived signed statement of identity that Jenkins can mint for one build. cosign hands that token to Fulcio, the Sigstore certificate authority (CA), and Fulcio returns a certificate good for only a few minutes. Long enough to sign one image. The signature and that certificate go into Rekor, a public transparency log, so anyone can confirm later that this identity signed this digest at this time. Nothing long-lived ever lands on disk.

# $OIDC_TOKEN is a short-lived identity token minted by Jenkins for this build.
# In cosign 2.x keyless is the default: no key flag means "go keyless".
cosign sign --yes \
  --identity-token "$OIDC_TOKEN" \
  registry.example.com/checkout@sha256:9f2a4c8e...b1

Generating ephemeral keys...
Retrieving signed certificate...
 
        Note that there may be personally identifiable information associated with this
        signed artifact. This information is stored in a public transparency log and
        cannot be removed later.
 
Successfully verified SCT...
tlog entry created with index: 74650912
Pushing signature to: registry.example.com/checkout

The key-based fallback: a key in the Jenkins safe

Keyless asks two things of your pipeline: that it can mint an OIDC token, and that it can reach the public Sigstore services. On a locked-down or air-gapped network you get neither, so you fall back to an old-fashioned key pair. You generate one with cosign. The public half can go wherever you like, pinned to a wiki page if that suits you. Treat the private half as the only key to a safe: it lives in the Jenkins credential store and is decrypted only inside the step that needs it. The withCredentials block loads the key file path and its password into environment variables that exist for that one sh step and are masked in the log, so the secret sits where cosign reads it and nowhere else.

stage('Sign (key-based)') {
  steps {
    withCredentials([
      file(credentialsId: 'cosign-key',  variable: 'COSIGN_KEY'),      // cosign.key file
      string(credentialsId: 'cosign-pass', variable: 'COSIGN_PASSWORD') // decrypts the key
    ]) {
      // cosign reads COSIGN_PASSWORD from the environment to unlock the key.
      // Neither the key path nor the password is ever echoed.
      sh 'cosign sign --yes --key "$COSIGN_KEY" registry.example.com/checkout@sha256:9f2a4c8e...b1'
    }
  }
}

An ingredients list, written by syft

An SBOM (Software Bill of Materials) is the ingredients list printed on the side of a cereal box. It names every package, library and version baked into your image. When a nasty bug lands in some popular library next month, you can answer "is that in anything we shipped?" in seconds instead of guessing. syft is the tool that reads an image and writes that list. Here it writes the list in SPDX (Software Package Data Exchange) format, a widely used standard for SBOMs, into a file you will attach to the image in the next step.

syft registry.example.com/checkout@sha256:9f2a4c8e...b1 \
  -o spdx-json=sbom.spdx.json

 ✔ Pulled image
 ✔ Loaded image                          registry.example.com/checkout@sha256:9f2a4c8e...
 ✔ Parsed image                          sha256:9f2a4c8e...
 ✔ Cataloged contents                    c0e6a2f4b8d1e3f5a7c9...
   ├── ✔ Packages                        [153 packages]
   ├── ✔ Executables                     [412 executables]
   ├── ✔ File metadata                   [2,341 locations]
   └── ✔ File digests                    [2,341 files]

Stapling the SBOM on, and sealing the staple

A plain sbom.spdx.json file sitting next to the image proves nothing. Anyone can hand you a made-up ingredients list. An attestation staples the list to the image and seals the staple. cosign attest wraps your SBOM in a signed statement saying that this identity vouches that this SBOM describes this exact digest, stores it in the registry alongside the image, and records it in Rekor. Swap the image and the attestation no longer matches. Edit the SBOM and the signature breaks. The signing works the same way as before: keyless with your OIDC token, or key-based with your key.

cosign attest --yes \
  --identity-token "$OIDC_TOKEN" \
  --predicate sbom.spdx.json \
  --type spdxjson \
  registry.example.com/checkout@sha256:9f2a4c8e...b1

Using payload from: sbom.spdx.json
Generating ephemeral keys...
Retrieving signed certificate...
Successfully verified SCT...
tlog entry created with index: 74651003

SLSA provenance: the chain-of-custody form

SLSA (Supply-chain Levels for Software Artifacts, said out loud as "salsa") provenance is the chain-of-custody form that travels with a shipment: which factory, which recipe, which raw materials, which batch. For a build that means which pipeline produced the image, from which git commit, at which build number. You write the facts into a small predicate file, and in a real pipeline your Jenkinsfile fills them in from build variables. Then you sign it onto the image the same way you signed the SBOM. A verifier can now confirm two separate things: that the image is yours, and that it came out of your build system rather than off someone's laptop.

{
  "buildDefinition": {
    "buildType": "https://jenkins.example.com/pipeline/v1",
    "externalParameters": {
      "repository": "https://git.example.com/checkout",
      "ref": "refs/heads/main"
    },
    "resolvedDependencies": [
      { "uri": "git+https://git.example.com/checkout@4b8c9d1",
        "digest": { "gitCommit": "4b8c9d1e2f..." } }
    ]
  },
  "runDetails": {
    "builder": { "id": "https://jenkins.example.com/job/checkout-build" },
    "metadata": { "invocationId": "build-482" }
  }
}

cosign attest --yes \
  --identity-token "$OIDC_TOKEN" \
  --predicate provenance.json \
  --type slsaprovenance1 \
  registry.example.com/checkout@sha256:9f2a4c8e...b1

Using payload from: provenance.json
Generating ephemeral keys...
Retrieving signed certificate...
Successfully verified SCT...
tlog entry created with index: 74651078

Verify before you deploy, and gate on it

A signature nobody checks is decoration, so the deploy stage becomes the bouncer on the door. cosign verify pulls down the signature and its certificate and confirms two things that you have to name out loud: that the signer's identity matches the pipeline you expect (--certificate-identity), and that the certificate came from the issuer you expect (--certificate-oidc-issuer). cosign 2.x makes both flags mandatory for keyless verification for exactly that reason. You cannot absent-mindedly accept a signature from any old signer, because you are the one naming who is allowed to sign. If you signed with a key instead, you pass --key cosign.pub here. A mismatch, a missing signature or a tampered image each make cosign exit with a non-zero code.

cosign verify \
  --certificate-identity   "https://jenkins.example.com/job/checkout-build/" \
  --certificate-oidc-issuer "https://jenkins.example.com/oidc" \
  registry.example.com/checkout@sha256:9f2a4c8e...b1

Verification for registry.example.com/checkout@sha256:9f2a4c8e...b1 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - Existence of the claims in the transparency log was verified offline
  - The code-signing certificate was verified using trusted certificate authority certificates
 
[{"critical":{"identity":{"docker-reference":"registry.example.com/checkout"},
  "image":{"docker-manifest-digest":"sha256:9f2a4c8e...b1"},"type":"cosign container image signature"},
  "optional":{"Issuer":"https://jenkins.example.com/oidc",
  "Subject":"https://jenkins.example.com/job/checkout-build/"}}]

stage('Verify & deploy') {
  steps {
    // cosign exits non-zero on any failure; a failed sh step aborts the stage,
    // so deploy.sh below is reached only when verification passed.
    sh '''
      cosign verify \
        --certificate-identity    "https://jenkins.example.com/job/checkout-build/" \
        --certificate-oidc-issuer "https://jenkins.example.com/oidc" \
        registry.example.com/checkout@sha256:9f2a4c8e...b1
    '''
    sh './deploy.sh registry.example.com/checkout@sha256:9f2a4c8e...b1'
  }
}

The ordering is the whole gate. A failing sh step stops the stage, so deploy.sh never runs unless cosign printed its green checks first. An image with no signature, a signature from the wrong identity, or a digest that does not match what was signed cannot slip past into production. The pipeline goes red and stops. Verify by digest here as well, the same digest you signed, so the thing you check is the exact thing you ship.

Try this

Work through “Verify before you deploy, and gate on it” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: keyless needs a real OIDC token, and a private key belongs in neither the image nor the logs. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related