Exceptions with expiry

Break-glass that closes itself.

A break-glass key is that little box on a hospital wall with a spare key behind a pane of glass. Anyone can smash it in an emergency, and that is the point. The glass is not the security. What protects the key is that breaking it is loud, it gets logged, and somebody replaces the pane the next morning. A policy exception should work the same way: cheap to get when you genuinely need one, impossible to keep quietly forever.

Refuse to grant exceptions at all and your policy program grows shadow forks. Teams stand up their own clusters, switch off the admission webhook (the cluster component that inspects every new object before the API server writes it to storage), or invent labels that mean "leave us alone". Grant exceptions that never end and you have written careful documentation for your own holes. The workable middle is a small number of explicit, reviewed, time-boxed exceptions with named owners. Every one should answer four questions on its face: why, who, until when, and what compensating control covers the gap while it is open.

What Happens When You Refuse to Grant Exceptions

A payments team has a vendor installer that formats a disk on first run. It genuinely needs a privileged container, meaning a container that runs with nearly all of the host's capabilities and sits roughly one step away from being root on the node itself. Your policy blocks privileged containers. The launch date is Friday. What happens next is boringly predictable. Somebody holding cluster-admin (the Kubernetes role that can do anything to anything) scales the policy engine deployment down to zero replicas for ten minutes, runs the installer, and scales it back up.

Those ten minutes are worse than any exception you could have written. Nothing records which rule was bypassed or which workload used the gap. Every other policy in the cluster was off during the same window, so whatever else got created in that period sailed through unchecked as well. And the habit is now established. The next team will not bother asking you.

An attacker who phishes a developer laptop and finds working cluster credentials is delighted by that pattern, because the bypass route is already normal behaviour that nobody reviews. A reviewed exception is the opposite: a named object living in Git, scoped narrowly enough to read in one breath, with a date on it. You are trading one small documented hole for the removal of a large undocumented one.

Write the Exception as Data, Not as a Slack Message

Kyverno models an exception as a PolicyException object. Gatekeeper does it with excludedNamespaces inside a Constraint's match block, or with entries in the cluster-wide Config object. Plain OPA (Open Policy Agent, the general-purpose policy engine that Gatekeeper wraps) does it with a data document that your Rego rules consult before deciding, Rego being the language OPA policies are written in. Use whichever mechanism your engine supports natively. An exception recorded as a commented-out rule, or as a chat thread somebody starred, is not an exception. It is a rumour.

Kyverno keeps this feature switched off until you ask for it. You set the container flag enablePolicyException to true on the admission controller, and you also set exceptionNamespace to name the one namespace where exception objects are allowed to live. That second flag matters more than it looks. Leave it unset and Kyverno honours PolicyExceptions from every namespace in the cluster, which means anyone with write access to their own namespace can mint their own exemptions and your policy engine quietly becomes a suggestion box.

apiVersion: kyverno.io/v2
kind: PolicyException
metadata:
  name: vendor-installer-privileged
  namespace: policy-exceptions      # must match the exceptionNamespace flag
  labels:
    # The self-closing part. Kyverno's cleanup controller deletes this
    # object once the date passes. Relative values like 72h also work.
    cleanup.kyverno.io/ttl: "2026-08-01"
  annotations:
    owner: payments-team
    ticket: SEC-1234
    expires-on: "2026-08-01"
    reason: "Vendor installer v4.2 formats the data volume on first run."
    compensating-control: "Namespace has a default-deny NetworkPolicy; the Job
      is triggered by hand and its node is drained afterwards."
spec:
  # Exempt admission only. Background scans keep evaluating the policy,
  # so this Pod still shows as a fail in the policy report.
  background: false
  exceptions:
    - policyName: disallow-privileged-containers
      ruleNames:
        - privileged-containers
        - autogen-privileged-containers
        - autogen-cronjob-privileged-containers
  match:
    any:
      - resources:
          kinds:
            - Pod
            - Job
          namespaces:
            - batch
          names:
            - "vendor-installer-*"

Three details in that file cause most of the real-world failures. The first is the autogen rule names. Kyverno automatically generates copies of a Pod rule for the controllers that create Pods, prefixing them with autogen- for Deployments and Jobs and autogen-cronjob- for CronJobs. List only the base rule name and your exception covers a bare Pod while silently missing the Pod that the Job actually creates. The second is the names field. A Job stamps out Pods with a random suffix, so an exact name never matches the child Pod, and that wildcard is doing load-bearing work. The third is background: false, which the middle of this lesson is built around. Leave it out and the exception applies during background scans too, flipping the result in your policy report from fail to skip and hiding the very workload you meant to keep watching.

kubectl apply -f policies/exceptions/vendor-installer.yaml

policyexception.kyverno.io/vendor-installer-privileged created

kubectl get policyexceptions -A

NAMESPACE           NAME                          AGE
policy-exceptions   gpu-driver-hostpath           31d
policy-exceptions   legacy-agent-hostpid          88d
policy-exceptions   vendor-installer-privileged   12s

Give the Exception a Clock That Winds Itself

Three names and three ages. No owner, no reason, no date. That listing is a large part of why exception debt hides so well: everything you carefully wrote is sitting right there inside the objects, and the default output shows none of it. Pull the metadata into columns you can read and share. This is the command to paste into your weekly review.

kubectl get policyexceptions -A -o custom-columns=\
'NS:.metadata.namespace,NAME:.metadata.name,OWNER:.metadata.annotations.owner,TICKET:.metadata.annotations.ticket,EXPIRES:.metadata.annotations.expires-on'

NS                  NAME                          OWNER           TICKET     EXPIRES
policy-exceptions   gpu-driver-hostpath           ml-team         SEC-1401   2026-09-15
policy-exceptions   legacy-agent-hostpid          platform-team   SEC-0912   2026-06-30
policy-exceptions   vendor-installer-privileged   payments-team   SEC-1234   2026-08-01

Read the middle row again. Its own annotation says it expired last month, and the object is still here. An annotation is a promise made to humans; the label is the part a machine acts on. Put the label on screen beside the promise, which the -L flag does by printing any label you name as its own column.

kubectl get policyexceptions -A -L cleanup.kyverno.io/ttl

NAMESPACE           NAME                          AGE   CLEANUP.KYVERNO.IO/TTL
policy-exceptions   gpu-driver-hostpath           31d   2026-12-01
policy-exceptions   legacy-agent-hostpid          88d   <none>
policy-exceptions   vendor-installer-privileged   12s   2026-08-01

Now the failures are obvious. The legacy agent exception never carried a time-to-live label at all, so nothing was ever going to remove it and its June date was decoration. The GPU driver exception has one, but the label says December while the annotation promises September. Both bugs are invisible in a code review that only reads the annotations, and both are ordinary enough that you will ship them yourself eventually.

The label doing the work is cleanup.kyverno.io/ttl (TTL is time to live, a countdown after which the object gets deleted). It is built into Kyverno's cleanup controller and needs no cleanup policy of its own. The value can be a relative duration counted from when the controller first sees the label, written as 5m, 4h or 1d, or an absolute date such as 2026-08-01. There is a timestamp form too, and it trips up anyone who has typed a normal ISO 8601 timestamp before: Kyverno's layout is 2026-08-01T003000Z, with no colons inside the time. The familiar 2026-08-01T00:30:00Z does not match it. The controller rechecks labelled objects on an interval set by ttlReconciliationInterval, one minute by default, so deletion lands shortly after the deadline rather than on the exact second.

Two operational catches are worth knowing before you rely on any of this. The cleanup controller can only delete what its service account is permitted to delete, and Kyverno ships least-privilege permissions on purpose. You extend them by creating a ClusterRole labelled rbac.kyverno.io/aggregate-to-cleanup-controller: "true", which Kyverno folds into the controller's role. The other catch is the nastier one: an unrecognised time format produces a warning, not a rejection. A typo like 2026-8-1, missing its zero padding, leaves the exception alive forever while looking perfectly correct in the pull request, and the only trace is a line in the controller's log. That is precisely the failure this lesson exists to prevent, which is why the check further down verifies the format rather than trusting it.

# the morning after 2026-08-01
kubectl get policyexceptions -n policy-exceptions vendor-installer-privileged

Error from server (NotFound): policyexceptions.kyverno.io "vendor-installer-privileged" not found

Keep the Alarm Wired While the Door Is Open

Here is the idea that separates a mature policy program from a checkbox one. An exception should reduce enforcement. It should never reduce visibility. A shop that props a fire door open for a delivery still leaves the camera pointed at it.

Gatekeeper makes that split easy to see, because its two jobs run as separate processes. The webhook decides admission in real time. Audit walks everything already in the cluster on a loop and writes what it finds into each Constraint's status. You can switch one off and keep the other.

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config            # Gatekeeper only reconciles a Config named "config"
  namespace: gatekeeper-system
spec:
  match:
    # Enforcement paused for this namespace. Audit keeps counting,
    # so the violations stay on the scoreboard.
    - excludedNamespaces: ["batch"]
      processes: ["webhook"]
 
    # Genuinely out of scope: control-plane namespaces, everything off.
    - excludedNamespaces: ["kube-system", "gatekeeper-system"]
      processes: ["*"]

The valid values for processes are audit, webhook, sync (which controls what Gatekeeper caches for policies that need to see other objects), mutation-webhook, and the wildcard, which covers today's processes and any added later. Excluding a namespace from webhook leaves audit reporting its violations. Gatekeeper's --exempt-namespace flag behaves the same way and only exempts admission, though it needs the namespace itself labelled admission.gatekeeper.sh/ignore before it takes effect. Reaching for the wildcard is the move that blinds you, because it drops the namespace out of the count as well as out of enforcement.

Kyverno's version of the same split is the background field from the exception file above. Left unset, an exception applies during background scans as well as at admission, and the resource's result in the policy report changes from fail to skip. The workload vanishes from your violation numbers on the same day you approved it. Setting background: false keeps admission exempt and leaves the report result at fail, so the count stays honest. One caveat: if the policy rule reads request-time information such as the requesting user, background scanning cannot evaluate it and the field has to be false regardless.

Be precise about Gatekeeper's enforcement modes too, since people mix them up constantly. On a Constraint, enforcementAction defaults to deny, which rejects the request outright. Set it to warn and the object is admitted while a warning rides back on the API response, so kubectl prints the message and the deploy still succeeds. Set it to dryrun and the violation is recorded by audit while nothing is ever blocked. An unset field prints as <none> in the listing below and still means deny, which is the most common misreading of this output.

kubectl get constraints -o custom-columns=\
'NAME:.metadata.name,ACTION:.spec.enforcementAction,VIOLATIONS:.status.totalViolations'

NAME                       ACTION    VIOLATIONS
psp-privileged-container   <none>    3
require-team-label         dryrun    41
block-hostpath-volumes     warn      7

Read that output as a work queue. The first Constraint is enforcing, and those three violations are workloads that predate the policy or live in a namespace you excluded from the webhook. The second is still in dryrun with forty-one violations, which describes a policy nobody has finished rolling out. The third warns on admission, so people are seeing the message and shipping anyway. None of those numbers would exist if you had exempted the namespaces with the wildcard.

Verify the Hole Is Exactly the Size You Think

Writing an exception and assuming it works is how you end up with one far wider than you intended, or one that never applied at all while the team quietly bypassed the engine anyway. Server-side dry run is the honest test. It is a rehearsal that goes through the real admission chain, collects a real verdict from the real policies, and stores nothing. Run it before you apply the exception, so you can see the denial you are about to make disappear.

kubectl run vendor-installer-probe --image=vendor/installer:4.2 \
  --namespace batch --dry-run=server --restart=Never \
  --overrides='{"spec":{"containers":[{"name":"vendor-installer-probe",
    "image":"vendor/installer:4.2","securityContext":{"privileged":true}}]}}'

Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/batch/vendor-installer-probe was blocked due to the following policies
 
disallow-privileged-containers:
  privileged-containers: 'validation error: Privileged mode is disallowed. The fields
    spec.containers[*].securityContext.privileged, spec.initContainers[*].securityContext.privileged,
    and spec.ephemeralContainers[*].securityContext.privileged must be unset or set
    to `false`. rule privileged-containers failed at path
    /spec/containers/0/securityContext/privileged/'

That is what a failed admission actually returns. Not an exit code, not a silent skip: an error from the API server naming the webhook that objected, the blocked resource, the policy, the rule, and the exact JSON path that offended. Apply the exception, run the identical command, and the verdict flips.

kubectl apply -f policies/exceptions/vendor-installer.yaml
# then rerun the same probe command

policyexception.kyverno.io/vendor-installer-privileged created
pod/vendor-installer-probe created (server dry run)

Now test the edges, which is the step people skip. Rerun the probe with a name outside your wildcard, then again in a neighbouring namespace. Both should still be denied. If a Pod called anything at all in namespace batch now sails through, your match block is looser than you believed, and you have handed a whole namespace a route to root on its nodes rather than exempting one installer.

Fail the Build on Expired Exceptions

The label handles the cluster. Git is the other half. Delete an exception from the cluster while its file still sits in your repository and the next sync from Argo CD or Flux, the GitOps controllers that continuously push whatever is in Git back into the cluster, will recreate it and quietly reopen a hole that already expired. A short check in CI (continuous integration, the automated checks that run against every pull request) closes that loop, and it catches the typo problem from earlier by insisting that the human-readable annotation and the machine-readable label agree.

#!/usr/bin/env bash
# Needs mikefarah's yq v4, the command-line YAML reader.
set -euo pipefail
 
today=$(date -u +%Y-%m-%d)
fail=0
 
for f in policies/exceptions/*.yaml; do
  ann=$(yq '.metadata.annotations["expires-on"] // "MISSING"' "$f")
  ttl=$(yq '.metadata.labels["cleanup.kyverno.io/ttl"] // "MISSING"' "$f")
 
  if [ "$ann" = "MISSING" ] || [ "$ttl" = "MISSING" ]; then
    echo "FAIL $f: needs both an expires-on annotation and a ttl label"
    fail=1
  elif [ "$ann" != "$ttl" ]; then
    echo "FAIL $f: annotation says $ann but the ttl label says $ttl"
    fail=1
  elif [[ ! "$ann" =~ ^[0-9]{4}-[0-9]{2}-[0-9]{2}$ ]]; then
    echo "FAIL $f: '$ann' is not YYYY-MM-DD, so the ttl will never fire"
    fail=1
  elif [[ "$ann" < "$today" ]]; then
    echo "FAIL $f: expired on $ann and is still in Git"
    fail=1
  else
    echo "ok   $f: expires $ann"
  fi
done
 
exit "$fail"

./scripts/check-exception-expiry.sh; echo "exit=$?"

FAIL policies/exceptions/gpu-driver-hostpath.yaml: annotation says 2026-09-15 but the ttl label says 2026-12-01
FAIL policies/exceptions/legacy-agent-hostpid.yaml: needs both an expires-on annotation and a ttl label
FAIL policies/exceptions/nfs-backup-privileged.yaml: expired on 2026-06-30 and is still in Git
ok   policies/exceptions/vendor-installer.yaml: expires 2026-08-01
exit=1

Every problem you saw in the cluster listing is caught here in a pull request instead, plus one the cluster could not show you: nfs-backup-privileged expired and was deleted by the cleanup controller weeks ago, yet its file is still in the repository waiting for the next sync to bring it back. The string comparison in that last branch works because ISO dates sort the same way alphabetically and chronologically, which is a large part of why the format was designed that way. Wire the script into the same pipeline stage that runs conftest, the command-line tool that evaluates OPA policies against config files, so your exceptions face the same gate your policies do. Nobody keeps a calendar reminder alive for eighteen months, and the month they forget is the month an auditor asks.

The Honest Trade-off

Time-boxed exceptions cost real effort, and pretending otherwise is how the practice gets abandoned around month four. Somebody reviews each request. Somebody owns the approver group and answers for it when the queue is slow. Every expiry that fires interrupts a team that would rather be shipping features, and they will resent the renewal ceremony, especially the third time they renew the same vendor exception. Expect that. Expect too that maybe a quarter of your exceptions describe permanent facts about your environment rather than temporary gaps, and those deserve a different answer: fix the policy or narrow its scope, instead of renewing forever.

The payoff is that your unknown risk drops close to zero. You can answer the question "where are we not enforcing this control, and why" with a Git query rather than from memory, and that is the difference between a policy engine and a dashboard. Compensating controls have to be real for the trade to hold. Extra monitoring, network isolation, a manual approval gate, a node that gets drained afterwards: those are controls. An exception whose compensation is "we will be careful" is a hole with paperwork stapled to it.

One more habit worth building. Publish the exception count to engineering leadership every month, sitting next to your CVE numbers (CVE stands for Common Vulnerabilities and Exposures, the public catalogue of known software flaws). "Fourteen open exceptions, four of them older than ninety days" reads as risk in a way that a YAML file never will, and invisible exceptions never get budget to remediate. When the same rule generates its fifth exception request, treat that as a signal about the rule. Sometimes the right fix is a clearer denial message or a narrower policy, not another hole.

Try This

In a lab cluster running Kyverno with policy exceptions enabled, apply a policy that blocks privileged containers and confirm the block with the server dry-run probe above. Then write a PolicyException carrying both an expires-on annotation and a cleanup.kyverno.io/ttl label set to 3m. Run the same probe and watch it pass. Wait four minutes, list your exceptions, and watch the object delete itself with nobody touching anything. Run the probe once more to prove enforcement came back.

For the second half, break it on purpose. Set the label to 2026-8-1 instead of 2026-08-01, reapply, and wait. The exception stays alive because the format was never recognised, and the cluster gives you no error to notice. Run the CI script against the file and watch it catch what the cluster did not.

Next: enforcement points, and how CI checks, admission control and audit stop being three separate arguments and start working as one system.

Takeaway

The trap worth remembering here: expiry closes the door, not the room. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related