Kyverno mutate, generate, and verifyImages

More than validate — fix and prove provenance.

An electrical inspector who can only condemn a building is useful exactly once. The inspector who also tightens the loose outlet, makes sure every new floor gets a smoke alarm before anyone moves in, and checks the electrician's licence number against the trade registry changes how the whole building gets built. Kyverno's four rule types do those four jobs. validate refuses bad resources. mutate patches them into shape on the way in. generate creates companion objects when something new appears. verifyImages checks that a container image was built by whoever you think built it, using a cryptographic signature rather than the logo painted on the side of the van. All four run at admission time, which means the Kubernetes API (application programming interface) server takes a request to create or change something, pauses before writing it down, and asks Kyverno for an opinion first.

The security payoff sits mostly in the last two. A registry prefix allowlist, meaning a rule that says "images must start with ghcr.io/acme/", stops a stranger pulling from Docker Hub and stops almost nothing else. Anyone who can push to your own registry, including whoever picks up a leaked CI (continuous integration, the system that builds and publishes your code) token, can push ghcr.io/acme/app:1.4.2 and your allowlist waves it straight through. verifyImages changes the question from "where did this come from" to "who signed it, and can they prove it."

Four Rule Types, Two Audit Switches

Audit versus enforce is where teams get caught out, so be precise about where the setting lives. On Kyverno 1.13 and newer it sits on the rule: spec.rules[*].validate.failureAction and spec.rules[*].verifyImages[*].failureAction, each taking Audit or Enforce. The old policy-wide spec.validationFailureAction still works and is deprecated, so if you inherit a repository full of it, expect to migrate. Under Audit, the offending resource is admitted and the failure is recorded in a PolicyReport, a Kubernetes object holding pass, fail, warn, error and skip counts. Under Enforce, Kyverno returns a denial, the API server never writes the object, and a person running kubectl sees the refusal in their terminal.

That last sentence has a trapdoor in it. Most Pods are not created by people. A Deployment hands the job to a ReplicaSet controller, so when Kyverno denies the Pod, nobody's terminal shows anything: the Deployment sits at zero available replicas and the actual message is parked in the ReplicaSet's events, where kubectl describe replicaset will show you a run of identical failures. Engineers waste whole afternoons on this, staring at a Deployment that looks healthy in kubectl get deploy and reports nothing wrong.

Here is the part that surprises people: mutate and generate have no failureAction at all. There is no audit mode for a write. A mutate rule that matches will patch the object, and a generate rule that matches will create the companion resource, the moment you apply the policy. Your dry run is a narrow match block and a throwaway namespace, not a configuration flag. Teams who assume "we rolled everything out in audit first" and then find every Pod in the fleet wearing a new label learned this the loud way.

Mutate: Defaults Nobody Has to Copy and Paste

Mutate rules are for the boilerplate nobody enjoys retyping: a team label so pages route to the right on-call rotation, a default memory limit, runAsNonRoot: true on a security context. The +() prefix is an add anchor, and Kyverno applies that field only if it is not already present in the incoming resource. That one character is what makes the rule idempotent, meaning running it twice produces the same result as running it once, and it is what stops you stamping over a value a developer set on purpose.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-labels
spec:
  rules:
  - name: label-pods
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaces: ["team-*"]      # scope tight: this is your only dry run
    mutate:
      patchStrategicMerge:
        metadata:
          labels:
            +(managed-by): kyverno     # add anchor: applied only if absent
  - name: default-memory-limit
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaces: ["team-*"]
    mutate:
      patchStrategicMerge:
        spec:
          containers:
          - (name): "*"                # conditional anchor: every container
            resources:
              limits:
                +(memory): 512Mi

kubectl apply -f mutate-labels.yaml
kubectl run demo --image=nginx:1.27 --restart=Never -n team-payments
kubectl get pod demo -n team-payments --show-labels

clusterpolicy.kyverno.io/add-default-labels created
pod/demo created
NAME   READY   STATUS    RESTARTS   AGE   LABELS
demo   1/1     Running   0          4s    managed-by=kyverno,run=demo

Within a single policy, mutate rules run top to bottom in the order you wrote them, which is how cascading mutations work: rule two can key off a label rule one added. Across separate policies you get no such guarantee, so two policies both patching /spec/containers/0/resources/limits/memory will fight, and which one wins is not a thing you want to be learning about from a production incident. Own a field in exactly one policy, and write down somewhere readable who owns it.

Admission-time mutation only touches objects on their way in. For resources already running, Kyverno gives you mutate.targets together with mutate.mutateExistingOnPolicyUpdate: true, both set on the rule since 1.13, which tells the background controller to go and patch matching objects whenever the policy is created or changed. That is a fleet-wide write triggered by a kubectl apply, so review the pull request that enables it accordingly. The safer pattern for anything you genuinely care about is a pair: a mutate rule that supplies the default, plus a validate rule with failureAction: Enforce that denies the resource if the field is still wrong. Mutation handles hygiene, validation draws the hard line. Document the pair, because a developer who watches a field change after kubectl apply deserves to know which policy did it. Never mutate away a security context field somebody set deliberately unless there is a written standard saying you own that field.

Generate: The Companion Resources Nobody Remembers

A brand new namespace is wide open. Any Pod in it can reach any other Pod in the cluster, because Kubernetes ships with an allow-all network posture and stays that way until somebody writes a NetworkPolicy, the object that restricts which Pods may talk to which. A generate rule is the building code that installs the fire door before the tenants arrive: the instant a Namespace appears, a default-deny NetworkPolicy appears inside it, so a container compromised in a new team's namespace cannot immediately start port-scanning everyone else. The same rule type handles the other things nobody remembers on day one, such as a ResourceQuota capping how much the namespace can consume before it starves its neighbours.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: add-default-netpol
spec:
  rules:
  - name: default-deny
    match:
      any:
      - resources:
          kinds: [Namespace]
    exclude:
      any:
      - resources:
          namespaces: [kube-system, kube-public, default, kyverno]
    generate:
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      name: default-deny
      namespace: "{{request.object.metadata.name}}"
      synchronize: true                    # Kyverno owns it from here on
      orphanDownstreamOnPolicyDelete: true # keep the NetworkPolicies if the policy goes
      data:
        spec:
          podSelector: {}     # every Pod in the namespace
          policyTypes:
          - Ingress
          - Egress

synchronize decides who owns that generated object for the rest of its life, and it is the field people skip past. Set it to true and Kyverno reverts hand edits, pushes policy changes downstream, and removes the NetworkPolicy if the namespace stops matching. Leave it false and you get fire-and-forget: the object is created once, then drifts, and eighteen months later nobody can tell you whether that NetworkPolicy is load-bearing or a fossil.

Now read the second flag in that file, because it catches people out. With synchronization on, deleting the policy deletes everything it generated. That is the default, orphanDownstreamOnPolicyDelete: false, and it means a rushed kubectl delete clusterpolicy add-default-netpol at 2am strips the default-deny NetworkPolicy out of every namespace in the fleet at once. Setting it to true retains them. Neither answer is correct for everybody. Decide which failure you would rather explain, then write the reason in the policy's annotations for the person who inherits it.

Generate rules run through Kyverno's background controller rather than the admission webhook, and that controller has its own ServiceAccount with deliberately narrow permissions. If it cannot create the kind you asked for, nothing is denied and nothing shows up in your terminal. The namespace is created, the NetworkPolicy is not, and the default-deny you believe is protecting that team quietly does not exist. Kyverno records the work in UpdateRequest objects in its own namespace, and that is where you look.

kubectl create namespace team-payments
kubectl get networkpolicy -n team-payments
kubectl get updaterequests -n kyverno

namespace/team-payments created
No resources found in team-payments namespace.
NAME       POLICY               RULETYPE   RESOURCEKIND   RESOURCENAME    RESOURCENAMESPACE   STATUS   AGE
ur-x4k2p   add-default-netpol   generate   Namespace      team-payments                       Failed   6s

An UpdateRequest can sit in Pending, Completed, Skip, or Failed. Failed with no NetworkPolicy in sight almost always means permissions, and kubectl describe prints the reason in the standard wording Kubernetes uses when RBAC (role-based access control, the system deciding which accounts may do what) turns something down.

kubectl describe ur ur-x4k2p -n kyverno | tail -n 8

Status:
  Message:  failed to create resource networkpolicies.networking.k8s.io is
    forbidden: User "system:serviceaccount:kyverno:kyverno-background-controller"
    cannot create resource "networkpolicies" in API group "networking.k8s.io" in
    the namespace "team-payments"
  State:    Failed
Events:     <none>

The fix is a supplemental ClusterRole carrying an aggregation label. Kubernetes folds any role wearing that label into the background controller's top-level role automatically, so you extend Kyverno's permissions without editing anything the next Helm upgrade will overwrite. Note the label names the background controller specifically: the admission controller and the reports controller have their own, and picking the wrong one leaves you staring at an identical error.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: kyverno:generate-netpol
  labels:
    rbac.kyverno.io/aggregate-to-background-controller: "true"
rules:
- apiGroups: ["networking.k8s.io"]
  resources: ["networkpolicies"]
  verbs: ["create", "update", "delete", "get", "list", "watch"]

Now the honest danger in that policy. A default-deny rule that includes Egress also blocks DNS (domain name system, how a Pod turns a name like payments-api into a numeric network address), and a namespace whose Pods cannot resolve names looks exactly like a broken application to the team that owns it. You have bricked their namespace with a security control they never asked for, which is how platform teams lose the argument for the next five controls. Ship the kube-dns egress allowance inside the same generated object, or split ingress-deny and egress-deny into two rollout waves and do ingress first.

verifyImages: Proving Who Built the Thing

Cosign, the signing tool from the Sigstore project, does not tuck a signature inside the image. It pushes a separate object into the same registry repository, tagged after the image's digest, which is the sha256: content hash uniquely identifying those exact bytes. Kyverno's verifyImages rule fetches that object during admission and checks it against the identity your policy named.

Keyless signing is the part worth understanding, because it removes the key distribution problem that sinks most signing projects before they start. Rather than a long-lived private key somebody has to store, rotate, and eventually leak, your CI job receives a short-lived certificate from Fulcio (Sigstore's certificate authority) bound to its own workload identity, signs with it, and the record goes into Rekor (Sigstore's public append-only transparency log). Your policy then asserts an identity instead of a key: this image must have been signed by the GitHub Actions workflow at this exact path, on this exact branch. A stolen registry credential does not get you that.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-acme-images
spec:
  rules:
  - name: check-signature
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaces: ["team-*"]
    verifyImages:
    - imageReferences:
      - "ghcr.io/acme/*"          # only what you actually sign
      failureAction: Audit        # start here; flip per namespace later
      mutateDigest: true          # default: rewrite the tag to the verified digest
      verifyDigest: true          # default: require a digest after mutation
      required: true              # default: every matching image must verify
      attestors:
      - count: 1
        entries:
        - keyless:
            subject: "https://github.com/acme/app/.github/workflows/release.yaml@refs/heads/main"
            issuer: "https://token.actions.githubusercontent.com"
            rekor:
              url: https://rekor.sigstore.dev

mutateDigest defaults to true, and it is doing more work than its name suggests. Kyverno verifies ghcr.io/acme/app:1.4.2, resolves that tag to the digest it verified, and rewrites the Pod spec to pin that digest before the object is stored. Without it you have a time-of-check-to-time-of-use gap: Kyverno checks the signature on whatever :1.4.2 points at right now, then the kubelet (the agent on each node that pulls images and starts containers) pulls :1.4.2 a few seconds later, and an attacker who can move that tag serves different bytes than the ones you verified. Tags are mutable pointers. Digests are the content. Verify the tag, run the digest.

To see what a real block looks like, flip that one rule to failureAction: Enforce in a lab namespace and deploy something unsigned.

kubectl run demo --image=ghcr.io/acme/app:1.4.2 --restart=Never -n team-payments

Error from server: admission webhook "mutate.kyverno.svc-fail" denied the
request: resource Pod/team-payments/demo was blocked due to the following
policies
 
verify-acme-images:
  check-signature: 'image verification failed for ghcr.io/acme/app:1.4.2:
    signature not found'

Read the webhook name in that error carefully. Image verification starts inside Kyverno's mutating webhook, usually registered as mutate.kyverno.svc-fail, because the rule has to rewrite the digest as part of doing its job. It runs a second time in the validating webhook to apply the required and verifyDigest checks. Engineers searching logs for validate.kyverno.svc when signature enforcement misbehaves find nothing at all and conclude the policy never loaded. Nothing was written either: there is no Pod, no partially created object, no record beyond the report entry and the webhook logs.

Two failure modes to plan for before anyone types Enforce. Kyverno needs pull credentials for the repository holding the signature, so a private registry with no secret wired in fails with an authorization error that reads confusingly like a missing signature; point the rule at one with imageRegistryCredentials.secrets, or give the controller a global secret at install time. And keyless verification needs outbound network access to Fulcio and Rekor: when that path is down, or a proxy sits in the way, admission fails closed and nobody deploys anything. Kyverno caches successful verifications for 60 minutes by default, which softens a two-minute blip and does nothing for a two-hour outage. Air-gapped clusters run their own Rekor mirror, or set rekor.ignoreTlog: true with ctlog.ignoreSCT: true and fall back to key-based attestors, accepting the weaker guarantee that comes with skipping the transparency log.

Test It Before the Fleet Does

Never iterate on a mutate rule by throwing manifests at a live cluster. The kyverno command-line tool evaluates policies against local files, and that is what belongs in your pull request checks. The fixture that matters most is the one that already carries the field, because that is the test proving the add anchor works and the rule leaves a developer's deliberate value alone.

kyverno apply mutate-labels.yaml --resource pod-already-labelled.yaml

Applying 2 policy rule(s) to 1 resource(s)...
 
mutate policy add-default-labels applied to team-payments/Pod/demo:
 
apiVersion: v1
kind: Pod
metadata:
  labels:
    managed-by: platform-team
    team: payments
  name: demo
  namespace: team-payments
spec:
  containers:
  - image: nginx:1.27
    name: app
    resources:
      limits:
        memory: 512Mi
---
 
pass: 2, fail: 0, warn: 0, error: 0, skip: 0

managed-by stayed platform-team, and the memory limit the fixture did not set was added. That is both anchors behaving. Wire this into continuous integration next to kyverno test, which runs a declarative fixture file so every policy travels with its own expected results. Keep the pull requests small while you are there. One change that adds label mutation, generates NetworkPolicies, and switches on signature verification is unreviewable, and when it breaks something at 4pm on a Friday you will not know which third to revert. One concern per policy, one policy per rollout wave, and version the policy files like the production code they are.

Once it is live, read PolicyReport objects rather than webhook logs. Since Kyverno 1.10 there is one report per resource, not one per policy, and the name is the resource's unique identifier rather than anything you can guess, which throws people who last used Kyverno a few versions back. Each row carries the resource kind and name plus the counts.

kubectl get policyreport -A

NAMESPACE       NAME                                   KIND         NAME                        PASS   FAIL   WARN   ERROR   SKIP   AGE
team-payments   1a4f0c31-9d02-4c7e-8b41-5f2c9a0d77e3   Pod          checkout-6d4f9c7b8-2xn4h    2      1      0      0       0      3d
team-payments   4c8b1e77-3a55-4f10-9d6b-0b7e21c4a9f2   Pod          ledger-77c9d5f4b6-lq8vt     3      0      0      0       0      3d
team-search     8f13d2a0-6c74-41ab-bf39-2d5e8c1470aa   Pod          indexer-5b8f6c99d4-t7wpz    3      0      0      0       0      3d

Per-resource rows are precise and useless for a fleet-wide answer, so roll them up. jq (a command-line tool for slicing JSON) turns thousands of reports into the one number you need, which is how many failures each rule is producing and therefore what enforcing it would break.

kubectl get policyreport -A -o json \
  | jq -r '.items[] | .results[]? | select(.result=="fail") | "\(.policy)/\(.rule)"' \
  | sort | uniq -c | sort -rn

      5 verify-acme-images/check-signature
      2 require-run-as-nonroot/check-securitycontext

Five failures under an Audit rule are five workloads that would have been rejected the moment you switch to Enforce. Find out whose they are and why they are unsigned before you flip anything, and alert on the error count as well as the fail count, because an erroring rule is not a passing rule. Team search is already clean in that listing, which is your argument for enforcing there first and letting payments catch up. Rolling out per namespace instead of per cluster is the whole reason failureAction moved onto the rule.

The API Under Your Feet Is Moving

One more thing you need to know before you write a hundred of these. Kyverno 1.17 marked ClusterPolicy deprecated, with removal planned around 1.20, and the replacement is a family of typed objects under policies.kyverno.io/v1: ValidatingPolicy, MutatingPolicy, GeneratingPolicy, ImageValidatingPolicy and DeletingPolicy, plus namespaced variants. They match resources the way Kubernetes' own admission policies do, with matchConstraints.resourceRules, and they express logic in CEL (Common Expression Language, the small expression language Kubernetes already uses for validating admission policies) rather than anchors and overlays.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
  name: verify-acme-images
spec:
  validationActions: [Audit]          # [Deny] is the new name for Enforce
  matchConstraints:
    resourceRules:
    - apiGroups: [""]
      apiVersions: ["v1"]
      resources: ["pods"]
      operations: ["CREATE", "UPDATE"]
  matchImageReferences:
  - glob: "ghcr.io/acme/*"
  attestors:
  - name: acmeci
    cosign:
      keyless:
        identities:
        - issuer: "https://token.actions.githubusercontent.com"
          subject: "https://github.com/acme/app/.github/workflows/release.yaml@refs/heads/main"
  validationConfigurations:
    mutateDigest: true
  validations:
  - expression: >-
      images.containers.map(image, verifyImageSignatures(image, [attestors.acmeci])).all(e, e > 0)
    message: "image signature verification failed"

Everything you have read still applies, which is the point of learning the behaviour rather than the field names. Writes have no audit mode in the new API either. Generated objects still outlive whoever created them. Digest pinning still closes the same gap, and spec.evaluation.mutateExisting.enabled on a MutatingPolicy is the same fleet-wide write with a different spelling. Write new policies against the new types, plan the migration of the old ones, and stop treating the version of Kyverno in your clusters as a detail somebody else tracks.

Try This

In a lab cluster, apply the label mutation, create a Pod, confirm the label landed, then delete the ClusterPolicy and create a second Pod. The first Pod keeps its label and the second never receives one, which is the whole behaviour of admission control demonstrated in four commands. If you want objects that already exist to be fixed too, that is mutate.targets plus the background controller, and it is a different risk conversation with a much larger blast radius.

kubectl get clusterpolicy
kubectl delete clusterpolicy add-default-labels
kubectl run demo2 --image=nginx:1.27 --restart=Never -n team-payments
kubectl get pods -n team-payments --show-labels

NAME                 ADMISSION   BACKGROUND   READY   AGE   MESSAGE
add-default-labels   true        true         True    12m   Ready
add-default-netpol   true        true         True    12m   Ready
clusterpolicy.kyverno.io "add-default-labels" deleted
pod/demo2 created
NAME    READY   STATUS    RESTARTS   AGE   LABELS
demo    1/1     Running   0          12m   managed-by=kyverno,run=demo
demo2   1/1     Running   0          3s    run=demo2

Every control here leaves behind a write that outlives the policy which made it, and some of them vanish the moment that policy does. So the interesting question stops being "does this rule work" and becomes "how do we version it, roll it out in waves, grant one team an exception without leaving a permanent hole, and retire it later without breaking the things it created." That is the policy lifecycle, and it is where we go next.

Takeaway

The trap worth remembering here: a mutation has no dry run and no undo. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related