Measuring policy effectiveness

Findings, SLOs, and when to retire a rule.

A town puts a speed camera outside a school and the crashes at that junction stop. Two years later there are forty cameras across the borough, four of them pointing at roads since turned into cul-de-sacs, and the only figure anyone reports upward is how many cameras are installed.

Policy programs age the same way. Rules pile up because adding one takes a pull request and removing one takes an argument. Teams learn which rules they can get an exception for and route around the rest. Two years in you have four hundred lines of Rego, the policy language used by Open Policy Agent (OPA, the general-purpose rules engine that Gatekeeper wraps for Kubernetes), and no way to answer the question that matters: which of these controls reduces risk?

An attacker does not care how many Constraints you have installed. They care whether the namespace they landed in is excluded from your admission webhook, the cluster component that gets shown every new object before the API server (the front door that every kubectl command and every controller talks to) writes it to storage. They care whether your audit loop ran today, and whether anyone would notice the difference between a policy enforcing and a policy switched off. Those things are measurable. Policy count is not, which is why it is the number most programs report.

Five measurements carry nearly all the signal, and plain tile names are fine: violations by rule, active exceptions, blocks in CI (continuous integration, the automated checks that run on every pull request), admission denies, and median hours to fix. Add a sixth once you can: how often a denial turns out to be the rule being wrong rather than the workload.

One rule about people before any of the numbers. Security and platform read the same dashboard in the same review. If only security watches violation counts, developers experience denials as weather: something that happens to them, with nobody to argue with. Shared ownership is what turns a deny into a bug report.

Zero Denies Is Two Different Facts

A smoke alarm that has not sounded in a year is telling you one of two opposite things. Either nothing burned, or the battery died in March and you have been sleeping under a dead box. Only the test button separates them. A deny counter sitting at zero for six weeks carries exactly the same ambiguity, and most teams read it as the good news.

The lesson on enforcement points covered why. Gatekeeper ships its main validating webhook, the one named validation.gatekeeper.sh, with failurePolicy set to Ignore. If those webhook pods are unreachable, the API server admits everything and writes a line in its own log saying it skipped the check. Nobody has to delete a rule to disable your program. Scaling the controller to zero, an expired serving certificate, or the ignore label sitting on a namespace all produce the same flat line.

So build a test button. Keep one manifest you know breaks a live rule, apply it on a schedule, and treat a successful apply as the alarm. Server-side dry run makes this safe to point at production: kubectl sends the object through the real admission chain on the API server, collects the verdict, and throws the object away instead of saving it.

apiVersion: v1
kind: Pod
metadata:
  name: policy-canary
  namespace: policy-canary
  labels:
    team: platform-security   # satisfies require-team-label on purpose, so
                              # only the rule under test can fail this Pod
spec:
  containers:
    - name: c
      image: nginx:1.27
      securityContext:
        privileged: true      # breaks psp-privileged-container

kubectl apply --dry-run=server -f canary/bad-pod.yaml

Error from server (Forbidden): error when creating "canary/bad-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [psp-privileged-container] Privileged container is not allowed: c, securityContext: {"privileged": true}

Look closely at what a refusal actually is, because the three enforcement modes return three different things and people conflate them constantly. A deny is an HTTP 403 Forbidden from the API server, carrying the constraint name in square brackets and whatever message your rule set. The object is never written. A warn admits the object normally and returns the message in a Warning response header, which kubectl prints on a line starting with "Warning:" that scrolls past in a pipeline nobody reads. A dryrun returns nothing at all to the person applying; the only trace is a row in the Constraint's status and a line in the audit log. If your dashboard treats those three as one number, it is measuring nothing.

out=/var/lib/node_exporter/textfile_collector/policy_canary.prom
{
  echo '# HELP policy_canary_admitted 1 if the known-bad Pod was admitted.'
  echo '# TYPE policy_canary_admitted gauge'
  if kubectl apply --dry-run=server -f canary/bad-pod.yaml >/dev/null 2>&1; then
    echo 'policy_canary_admitted 1'   # the apply SUCCEEDED, which is the alarm
  else
    echo 'policy_canary_admitted 0'
  fi
} > "$out.tmp" && mv "$out.tmp" "$out"
cat "$out"

# HELP policy_canary_admitted 1 if the known-bad Pod was admitted.
# TYPE policy_canary_admitted gauge
policy_canary_admitted 0

Write to a temporary file and rename it, because a scraper that catches you mid-write reads a truncated file and reports garbage. A deliberately broken Pod refused every five minutes proves more than any count of policies shipped. It is also the only measurement in this lesson that is live rather than sampled, which makes it the thing to page on at three in the morning.

Read The Numbers Off The Cluster First

Before any dashboard exists, get the raw counts. Gatekeeper's audit controller sweeps every object in the cluster on a timer and writes what it finds into each Constraint's status field.

kubectl get constraints -o custom-columns=\
'NAME:.metadata.name,ACTION:.spec.enforcementAction,VIOLATIONS:.status.totalViolations'

NAME                       ACTION    VIOLATIONS
block-hostpath-volumes     warn      7
psp-host-network           deny      0
psp-privileged-container   <none>    3
require-memory-limits      <none>    12
require-team-label         dryrun    41

conftest test --policy policy/ --output json rendered/ > ci-results.json
jq -r '[.[] | .failures[]?.msg] | group_by(.) | map("\(length)\t\(.[0])")[]' ci-results.json

Read that as a work queue, not a report card. An unset ACTION still means deny, which is the most common misreading here and the one that makes compliance reports understate their own enforcement surface. The bottom row is a policy nobody finished rolling out: forty-one violations parked in dryrun, where audit records the problem and nothing is blocked. The warn row admits the object and returns that warning header, so seven teams read the message and shipped anyway. The two deny rows still show violations because audit checks objects that already exist, and denial only ever stopped new ones: those twelve missing memory limits predate the rule, or landed while the webhook was down. The zero on psp-host-network is the genuinely ambiguous one, and the retirement section resolves it.

A single global figure hides the one rule burning one product org, so break it down by rule and namespace using jq, the command-line tool for slicing JSON.

kubectl get constraints -o json | jq -r '
  .items[] as $c
  | ($c.status.violations // [])[]
  | "\($c.metadata.name)\t\(.namespace)"' \
  | sort | uniq -c | sort -rn

     12 require-team-label	payments
      9 require-memory-limits	batch
      8 require-team-label	search
      5 block-hostpath-volumes	ml
      3 require-memory-limits	payments
      3 psp-privileged-container	batch
      2 block-hostpath-volumes	batch

Do the arithmetic before you trust it. The require-memory-limits rows add to twelve, matching its total. The require-team-label rows add to twenty, and its total was forty-one. Gatekeeper caps the detailed status.violations list at whatever --constraint-violations-limit says, twenty by default, while status.totalViolations stays uncapped. Your chart is a truncated sample of at most twenty rows per rule, and the truncation is not random. It is whatever audit happened to visit first, so teams whose namespaces sort late look spotless when they are anything but.

Three ways out, each with a price. Raise the limit and every Constraint object grows, which matters because these objects live in etcd (the key-value database holding all cluster state), where a single object has a default ceiling around 1.5 MiB. Turn on --emit-audit-events so each violation becomes a Kubernetes Event carrying its own timestamp, the cheapest honest route to a time-to-remediate figure; Events are deleted after an hour by default, so treat that as a pipe into a log store rather than the store itself. Or turn on violation export, which publishes audit results to a message sink instead of stuffing them into object status. Export is still moving between alpha and beta across releases, so read the flags for the version you actually run.

Send each team its own rows monthly, and send help with them, because a top-offenders list arriving with no offer to fix the charts produces exception requests rather than fixes. Kyverno holds the same information differently: since version 1.10 it writes one PolicyReport per resource rather than one per namespace, so the counts are complete rather than capped, at the cost of a great many small objects. Cluster-scoped results land in clusterpolicyreports instead, which the command below will miss.

kubectl get policyreports -A -o json | jq -r '
  [.items[].results[]? | select(.result == "fail")]
  | group_by(.policy)[]
  | "\(length)\t\(.[0].policy)"' | sort -rn

118	require-run-as-nonroot
44	disallow-privilege-escalation
9	require-pod-requests-limits

Turn Counts Into Time Series Without Fooling Yourself

A fuel gauge and an odometer both fail, but in opposite directions. The odometer only climbs, so when the car stops the number stops and you can see that it stopped. The fuel gauge shows a level, and a broken one shows the last level it knew, forever, looking entirely plausible. Gatekeeper's violation metric is a fuel gauge.

kubectl -n gatekeeper-system port-forward deploy/gatekeeper-audit 8888:8888 >/dev/null &
curl -s localhost:8888/metrics | grep -E '^gatekeeper_(violations|audit_last_run_time)'

gatekeeper_violations{enforcement_action="deny"} 15
gatekeeper_violations{enforcement_action="dryrun"} 41
gatekeeper_violations{enforcement_action="warn"} 7
gatekeeper_audit_last_run_time 1.785110412e+09

Two details there earn their keep. First, which pod you scraped. gatekeeper_violations comes from the audit deployment, not the webhook pods, so a scrape config that only targets the controller manager hands you a dashboard with no violations on it and no error explaining why. Second, the label set is enforcement_action and nothing else, so per-rule numbers never come from this metric. The admission side lives on the other deployment entirely.

kubectl -n gatekeeper-system port-forward deploy/gatekeeper-controller-manager 8889:8888 >/dev/null &
curl -s localhost:8889/metrics | grep '^gatekeeper_request_count'

gatekeeper_request_count{admission_status="allow"} 1842203
gatekeeper_request_count{admission_status="deny"} 96

Track that as a ratio rather than a raw count, which moves whenever deploy volume moves. Metric names and label values have shifted across Gatekeeper releases, and some builds append _total to counters, so grep the endpoint on the version you run before you write PromQL (the query language Prometheus uses) against a name you half remember from a blog post.

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: policy-program
  namespace: monitoring
spec:
  groups:
    - name: policy-program.rules
      interval: 1m
      rules:
        # What fraction of policy failures are caught before the cluster?
        # policy_ci_failures_total is emitted by your CI job. No policy
        # engine can see inside your pipeline, so you publish this one.
        - record: policy:shift_left_ratio:7d
          expr: |
            sum(increase(policy_ci_failures_total[7d]))
            /
            clamp_min(
                sum(increase(policy_ci_failures_total[7d]))
              + sum(increase(gatekeeper_request_count{admission_status="deny"}[7d]))
            , 1)
 
        # The measurement going dark. Page on this, not on the count.
        - alert: PolicyAuditStale
          expr: time() - gatekeeper_audit_last_run_time > 900
          for: 10m
          labels:
            severity: page
          annotations:
            summary: "Audit has not run for 15m; every violation gauge is frozen"
 
        # The control itself. A successful apply of a known-bad Pod pages.
        - alert: PolicyCanaryAdmitted
          expr: max_over_time(policy_canary_admitted[15m]) > 0
          for: 5m
          labels:
            severity: page
          annotations:
            summary: "Known-bad Pod was admitted; enforcement is not running"
 
        # Error budget for a policy rollout. 0.5/s is 30 denies a minute.
        - alert: PolicyRolloutBurningBudget
          expr: sum(rate(gatekeeper_request_count{admission_status="deny"}[10m])) > 0.5
          for: 15m
          labels:
            severity: page
          annotations:
            summary: "Sustained deny spike; roll back the last policy bundle"

That last rule is an error budget for policy changes, borrowed from the way reliability teams budget failure for services. A rollout that starts blocking deploys spends the budget, and spending it triggers a bundle rollback rather than a debate at four o'clock on a Friday. Set the threshold from your own baseline, not from the number written above.

The Parity Ratio That Tells You If Shift-Left Is Real

A goalkeeper making forty saves in a match is not evidence of a great goalkeeper. It is evidence of a defence that stopped working. Admission control is the goalkeeper, and every deny it produces is a problem that got all the way to the cluster before anything objected.

Compare the two gates over one window. If admission catches most of the failures, shift-left is failing: developers are learning at deploy time about problems a branch check could have shown them in eight seconds. The usual cause is that CI evaluates different inputs from the cluster. A chart's values file can look compliant while the rendered Deployment is not, because a subchart default injects the offending field during templating. Render first, test the rendered YAML (the indented text format Kubernetes manifests are written in), and pin both sides to one policy bundle version.

If CI catches everything and admission has denied nothing in a month, hold the celebration. A green pipeline sitting beside a dead webhook looks identical to a green pipeline sitting beside a working one, and the canary is the only thing that tells them apart. No policy engine can see inside your pipeline, so the CI half of this ratio is a number you emit yourself. Conftest, the tool that runs Rego against local files, produces it in machine-readable form.

2	container must not run as root
8	container must set resources.limits.memory
4	image must come from registry.internal.example.com

Fourteen problems stopped in pull requests this week. If admission denied sixty-one in the same window, your catch rate is fourteen out of seventy-five, or nineteen percent, and four in every five failures reach a real cluster before anything says no. That percentage, not the deny count, is the honest reading on shift-left. If you push the CI number through a Prometheus pushgateway, remember that a push overwrites whatever shares its grouping key rather than adding to it, so a naive job reports the last run instead of the week.

Exception Debt Is A Number, Not A Feeling

The previous lesson made exceptions self-closing with a time-to-live label, TTL being a countdown after which Kyverno's cleanup controller deletes the object. Measurement tells you whether that actually held, because a mistyped date leaves an exception alive forever while looking perfectly correct in review.

kubectl get policyexceptions -A -o json \
  | jq -r --arg today "$(date -u +%F)" '.items[]
      | (.metadata.annotations["expires-on"]) as $e
      | [ (if $e == null then "NO-DATE" elif $e < $today then "EXPIRED" else "ok" end),
          ($e // "none"),
          (.metadata.namespace + "/" + .metadata.name),
          (.metadata.annotations.owner // "UNOWNED") ]
      | @tsv' | sort

EXPIRED	2026-06-30	policy-exceptions/legacy-agent-hostpid	platform-team
ok	2026-08-01	policy-exceptions/vendor-installer-privileged	payments-team
ok	2026-09-15	policy-exceptions/gpu-driver-hostpath	ml-team

Three exceptions, one of them nearly a month past its own stated date and still sitting in the cluster because it never carried the TTL label that would have removed it. That number belongs in front of engineering leadership monthly, beside your vulnerability counts. "Three open, one expired, owner named" reads as risk in a way that a directory full of YAML never will.

Watch the trend harder than the level. Exception count climbing while remediation stays flat means the program has started negotiating rather than enforcing: permission is moving faster than repair. When that shape holds for two months, pause new rule rollout until the queue clears. Keep exception debt on the same page as violation count, or teams will turn a red bar green by filing exemptions and nobody will see the trade they made.

Retiring A Rule Without Losing The Control

Removing a control is a security decision and deserves the same care you gave adding it. Two reasons justify it. Architecture removed the risk, and you can say why in one sentence: no workload mounts a host path because every node runs a locked-down runtime class. Or the rule maps to no current threat and makes only noise. A rule that is hard to understand but still covers real risk is a documentation problem, not a retirement candidate.

Neither reason justifies deleting anything on the day you decide. Run a watch period covering your slowest release cycle, with ninety days as a sensible floor, and run it in dryrun so you stop blocking while you keep counting.

kubectl patch k8spsphostnetworkingports psp-host-network --type=merge \
  -p '{"spec":{"enforcementAction":"dryrun"}}'

k8spsphostnetworkingports.constraints.gatekeeper.sh/psp-host-network patched

This is what resolves the ambiguity from the scoreboard earlier. Zero violations in deny mode told you very little, because deny mode stops new violating objects from ever being written, so the audit sweep has nothing left to count. Zero after ninety days in dryrun is real evidence: anything breaking that rule would have sailed straight through and been recorded, and nothing appeared.

kubectl get k8spsphostnetworkingports psp-host-network \
  -o jsonpath='{.status.totalViolations}{"\n"}'

0

One check before you delete anything. Find out who else built a Constraint from the same ConstraintTemplate, because that answer decides how much you are allowed to remove.

kubectl get constraints -o json \
  | jq -r '.items[] | select(.kind == "K8sPSPHostNetworkingPorts") | .metadata.name'

psp-host-network
psp-host-network-pci

kubectl delete k8spsphostnetworkingports psp-host-network

k8spsphostnetworkingports.constraints.gatekeeper.sh "psp-host-network" deleted

Record why in the same commit: the risk it covered, what removed that risk, how long the watch ran, and which control covers the gap now. A rule replaced by a stricter rule is a cleanup. A rule replaced by an assumption is a regression with a changelog entry. List retired rules beside the new ones in your reporting, because a culture that only rewards adding controls ends up with four hundred of them and no idea which twelve are load-bearing.

The Honest Trade-off

None of this is free. Label cardinality, meaning the number of distinct label combinations your metrics produce, grows faster than people expect: twenty-four constraints across three hundred namespaces is seven thousand two hundred separate time series before you add a single other label, and splitting by enforcement action triples that. Keep the per-namespace detail in logs and only the aggregates in metrics. Time to remediate needs an event pipeline you build and then keep alive. The false-positive rate is a human judgement, usually made by whoever wrote the rule and therefore generous. A program nobody reads is worse than no program, because it manufactures confidence without adding information.

Two habits cover what a time series cannot. Every quarter, ask three developers to describe the last policy denial they hit and what they did next. Half the time the message was clear and the fix took ten minutes. The other half, they copied a workaround out of a teammate's chart without understanding it, so the risk is still there wearing compliant clothing. Feed what you hear back into your denial messages.

Archive a snapshot of every number monthly. Auditors want evidence that a control operated across a period, not a screenshot of this morning, and twelve snapshots showing exception debt falling beats a dashboard nobody can rewind. Report the whole set in the same operational review as your platform SLOs (service level objectives, the targets you hold yourself to for things like availability). When leaders see deny rates and exception age sitting beside uptime, cleanup work gets funded.

Try This

Pick one Constraint that is currently denying and pull a week of its refusals. Classify ten of them by hand into true positive, where the workload really is wrong; false positive, where the rule is wrong; and exception-needed, where the workload is right for a reason the rule cannot see. Ten is enough. You are sampling, not auditing.

printf '%s\n' 'TP: 6' 'FP: 2' 'exception: 2' \
  'action: tighten message + fix two app charts'

TP: 6
FP: 2
exception: 2
action: tighten message + fix two app charts

The split is the point. Six true positives, two false positives and two exception requests describes a rule worth keeping with a wording fix. Two true positives and eight false positives describes a rule that is training an entire org to ignore your denials. If you cannot classify a denial in under a minute from the log line alone, the log line is missing a field.

If you build only two things this quarter, build the canary and the CI catch rate. The first tells you the control is alive. The second tells you whether admission is your last line of defence or your only one. Run the classification on your three noisiest rules every quarter and you will retire more policy than you write.

Takeaway

The trap worth remembering here: a frozen gauge looks exactly like a clean cluster. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related