Enforcement points

The same policy in CI, admission, and audit.

An international airport checks your passport more than once. Once when you book the ticket. Again at check-in, again at the security line, again at the boarding gate, and sometimes once more when you land. No single desk is trusted to be the *only* check, because every desk can be skipped on the right day: online check-in, a rushed gate agent, a systems outage. Policy-as-code works the same way. An enforcement point is any place in your delivery pipeline where something asks "is this allowed?" and then *acts on the answer*. A merge gets blocked. An API request gets rejected. A violation shows up as an alert.

The names for the moving parts come from access-control research, and they are less frightening than they sound. The policy decision point (PDP), the part that decides, is the engine that reads your rules and answers allow or deny. That is OPA (Open Policy Agent) or Kyverno, both covered in depth by earlier lessons. The policy enforcement point (PEP), the part that acts, is whatever intercepts an action, asks the decision point, and then does what it is told. Splitting those two apart is the whole trick. You author and test one set of rules, then enforce them in many places. What remains is choosing the *where*, and wiring it up.

Why one checkpoint is never enough

Every enforcement point, used on its own, has a bypass that people already know about. Enforce only in CI (continuous integration, the pipeline that builds and tests every change), and anyone holding cluster credentials can kubectl apply a manifest that never went near that pipeline. So can a stolen CI token, a GitOps controller syncing an unreviewed branch, or an engineer improvising at 3am during an incident. Enforce only at admission, and developers hear nothing until deploy time, which is slow and expensive to fix. The thousands of resources admitted *before* your policy existed never get re-checked, and a webhook outage quietly becomes a policy outage. Enforce only with runtime audit, and you spend your week writing incident reports instead of preventing incidents.

Production setups therefore stack three layers. CI validation on every pull request. Admission control at the cluster API (application programming interface) server, the front door every Kubernetes request has to walk through. And a continuous audit loop over whatever is already running. The pattern travels well beyond Kubernetes: conftest will evaluate a Terraform plan in JSON form as happily as it evaluates a manifest, and OPA can sit behind Envoy's ext_authz filter to approve or reject service-to-service traffic. Build the three-layer Kubernetes version first.

Layer 1: fail fast in CI

conftest is a command-line tool that runs Rego policies against files sitting on your disk. YAML and JSON config, HCL (HashiCorp Configuration Language, the language Terraform is written in), Dockerfiles. No cluster involved at any point. It is the cheapest enforcement point you will ever own, because feedback arrives in seconds, on the developer's own branch, while nothing yet exists that would need rolling back. Pull down the versioned policy bundle your team publishes, then run it over the rendered manifests.

$ conftest pull oci://ghcr.io/acme/policy-bundle:v1.4.2   # downloads into ./policy/
$ conftest test deploy/api.yaml -p policy/
FAIL - deploy/api.yaml - main - Deployment "api": container "api" must set resources.limits.memory
FAIL - deploy/api.yaml - main - Deployment "api": set securityContext.runAsNonRoot=true
 
6 tests, 4 passed, 0 warnings, 2 failures, 0 exceptions

A failure exits non-zero, so a required CI job blocks the merge with no extra glue on your side. Be honest about what this layer actually covers, though. It sees only the changes that travel *through the pipeline*. Treat CI enforcement as a developer-experience feature, the fastest feedback loop you have, and never as your security boundary.

Be honest about the rules themselves too. conftest collects deny rules from a package like main, and each document in the file arrives as input. Gatekeeper, which is OPA packaged as a Kubernetes admission controller and the subject of the next section, collects violation rules from inside a ConstraintTemplate, and the object arrives as input.review.object. Same intent, two different shapes, and nothing on the Gatekeeper side answers to conftest pull, because constraint templates ship as cluster resources rather than as a bundle you fetch. Generating both from one source and proving they still agree is a job somebody has to own, and it is the first thing that rots when nobody does. Kyverno is the stack where that parity comes for free, because kyverno apply in CI evaluates the same policy YAML the cluster admits with.

Layer 2: admission control, the only gate that answers in real time

When any client creates or changes an object, the Kubernetes API server runs it through authentication, authorization and mutation, then packs the request into an AdmissionReview JSON document and POSTs it over TLS (Transport Layer Security, the encryption behind the padlock in your browser) to every registered validating webhook whose rules match. At this layer, Gatekeeper and Kyverno are HTTPS servers that answer allowed: true/false before a deadline expires. The answer comes back while the request is still waiting, which is what *synchronous* means here, and that is why this is the only place that can stop an object from ever existing, whoever created it and by whatever route. The same power is why its failure behavior deserves a hard look.

$ kubectl apply -f deploy/api.yaml
Error from server (Forbidden): error when creating "deploy/api.yaml":
admission webhook "validation.gatekeeper.sh" denied the request:
[require-memory-limits] Deployment "api": container "api" must set resources.limits.memory
 
# The webhook's failure mode is configuration, not code — inspect it:
$ kubectl get validatingwebhookconfiguration gatekeeper-validating-webhook-configuration \
    -o jsonpath='{range .webhooks[*]}{.name}{"\t"}{.failurePolicy}{"\t"}{.timeoutSeconds}{"\n"}{end}'
validation.gatekeeper.sh	Ignore	3
check-ignore-label.gatekeeper.sh	Fail	3

Layer 3: audit what admission missed

The audit loop goes back over objects that already exist and re-runs the policies against them. Gatekeeper's audit controller sweeps the cluster every 60 seconds by default and writes what it finds into each constraint's status. That picks up the four populations admission structurally cannot see: resources created before the policy existed, resources admitted while the webhook was down or set to Ignore, resources sitting in exempted namespaces, and violations you introduced by *changing the policy itself*.

$ kubectl get constraints
NAME                                                                    ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8srequiredresources.constraints.gatekeeper.sh/require-memory-limits   deny                 12
 
$ kubectl get k8srequiredresources require-memory-limits -o json | jq '.status.violations[0]'
{
  "enforcementAction": "deny",
  "group": "apps",
  "kind": "Deployment",
  "message": "Deployment \"legacy-worker\": container \"worker\" must set resources.limits.memory",
  "name": "legacy-worker",
  "namespace": "batch",
  "version": "v1"
}

There is a ceiling here worth knowing about. Constraint status keeps a capped list of violations, 20 by default and tunable through --constraint-violations-limit, so at any real scale you read audit results from the exported Prometheus metrics or a violation-export sink rather than scraping status fields.

Hardening and what breaks at scale

Admission webhooks sit on the API server's hot path, so every hardening choice is really a latency choice. Keep timeoutSeconds at 3, and never push it past 10, because one slow webhook stalls *every* matching request across the whole cluster. Scope your webhook rules tightly: intercepting high-churn objects like events or leases multiplies webhook QPS (queries per second) and buys you no policy value whatsoever. Watch apiserver_admission_webhook_admission_duration_seconds for creep as your policy count grows. Then instrument the *relationship between the layers*. If audit violations climb while CI failures and admission denials stay flat, something is deploying around your pipeline. That divergence is your bypass detector, and it only reads cleanly while all three layers are enforcing the same rule set.

The problems waiting after this one are organizational rather than technical. Getting a single bundle onto fifty clusters without a flag day. Handing out a time-boxed exception that expires on its own instead of quietly becoming permanent. Deciding who owns a policy at the moment it blocks somebody else's deploy on a Friday afternoon. Those are the scale problems the rest of this course takes on.

Try this

Run these in a lab or a throwaway environment, so you see the real shape of the output instead of a screenshot from someone's blog. Point -p at whichever policy directory you already have, and use any manifest that breaks one of its rules.

conftest test deploy/api.yaml -p policy/ || true
kubectl auth can-i create pods --as=system:serviceaccount:default:default

FAIL - deploy/api.yaml - main - Deployment "api": container "api" must set resources.limits.memory
FAIL - deploy/api.yaml - main - Deployment "api": set securityContext.runAsNonRoot=true
 
6 tests, 4 passed, 0 warnings, 2 failures, 0 exceptions
no

Read the two answers together. conftest test exits non-zero the moment anything fails, which is what the || true swallows so the second command still gets to run. The FAIL lines are Layer 1 doing its job on your branch, before anything exists to roll back. The no says that service account cannot create pods at all, so it has no route around your pipeline. Now run the same check against an identity that answers yes, your own user for instance, and you have found a way into the cluster that CI never sees. That is the gap Layer 2 is there to close.

Takeaway

One decision should be able to fail a pull request, deny an admission, and turn up in an audit report. Many enforcement points, one policy brain.

Next: CI parity with Conftest and opa eval, so developers see failures before the cluster does.

Related