Your require-team-label constraint has been sitting in warn for a week, and Gatekeeper has denied exactly zero admission requests in that time. What actually tells you it is safe to flip it to deny?

The audit controller's totalViolations count for objects already in the cluster is zero, and has stayed there through the soak. The audit controller rescans what is already running, and only a zero count there protects you when a node drain or eviction recreates a violating pod and admission rejects it, turning maintenance into an outage.

Your bundle server is down for an hour. Some OPA engines have been running for weeks; a node that came up during the outage is starting a fresh one. What happens?

The long-running engines carry on with the last good bundle, but the fresh one has no policy at all until its first successful pull. That asymmetry is precisely why the bundle server is a tier-0 dependency. A failed refresh is a minor inconvenience; a cold start with nothing to fetch leaves an engine with no rules to apply.

Quarterly review. You query decision logs for the rule that requires images to come from your internal registry, and it has not fired once in 90 days. What do you do next?

Treat it as a repeal candidate and run the rollout backwards, deny to warn, then delete, so anything quietly leaning on it surfaces before the rule is gone. Ninety days of silence in the decision logs is the repeal signal, and reversing the ladder is how you find out in the open whether anything was depending on the rule.

Courses Policy-as-code at scalePolicy lifecycle

Policy lifecycle

Versioning, rollout waves, ownership.

Advanced30 min · lesson 8 of 13

A city that wants a new speed limit does not switch the cameras on overnight. Someone drafts the rule. The council argues about it. It gets published, posted on warning signs for a month, and only after all that does anyone get a ticket. Years later, when the road is rebuilt, the rule gets repealed. A policy in your infrastructure is the same kind of object: a rule a machine enforces on your behalf, like *every container image must come from our internal registry*. Writing the rule is the easy bit. The dangerous bit is everything wrapped around it. How it gets reviewed, how it reaches the engines that enforce it, how you switch it on without breaking production, and how you eventually kill it off.

That wrapping is the policy lifecycle: the managed path a rule walks from proposal to retirement. Author, review, test, package, distribute, audit, enforce, monitor, retire. Policy-as-code keeps your rules as plain files in Git and hands them to an engine such as OPA (Open Policy Agent, the open-source engine that answers allow-or-deny questions), which makes shipping a rule a software delivery problem. Every habit you already have for application code carries over. Version it, run it through CI (continuous integration, the automated build-and-test pipeline), roll it out in stages, watch what it does. Earlier lessons covered how an engine reaches a decision and how to test Rego (the small language you write OPA rules in). This one covers the gap between git merge and a request being turned away in production.

What goes wrong when you skip it

Skip the lifecycle and you fail in one of two opposite directions. The first is big-bang enforcement. A policy merges straight into deny mode. A new *every workload needs a team label* rule looks harmless in review, but 137 workloads already running do not carry that label. Every redeploy is blocked. The platform team spends the night hand-writing exemptions. The second failure is policy rot, the mirror image. Rules park in audit-forever mode, exemptions never expire, and nobody can remember why a given rule exists. Enforcement in name only. There is a security angle here too, and it is the one people miss. The channel you use to ship policy is an attack path. Whoever can serve policy to your engines gets to decide what is allowed into your clusters, so an unsigned bundle fetched over plain HTTP (the unencrypted version of the web protocol, which anyone sitting on the network path can quietly rewrite) is close to handing out cluster-admin, meaning full control of the cluster. Signing, versioning and staged rollout are what stop that, and all three live in the lifecycle rather than in the engine.

Packaging: bundles, revisions and signatures

OPA does not read Git. It eats a bundle, which behaves like a sealed shipping crate with a packing slip taped to the lid. Inside the crate: a gzipped tarball (one compressed archive file, the same idea as a .zip) holding your .rego files, any JSON data they need (JSON, JavaScript Object Notation, the plain-text format your data files are written in), and a .manifest. The packing slip declares a *revision*, which is a version string you choose, and *roots*, the namespaces this bundle owns so two bundles cannot silently overwrite each other's rules. Your CI job builds the crate, seals it with a private signing key, and pushes it to a bundle server. Any static HTTP server or OCI registry works (OCI, Open Container Initiative, the same sort of registry that already stores your container images). Put both the semantic version and the Git SHA in the revision string, where SHA is the short fingerprint Git stamps on every commit. That string is what your engines report back to you later, and you want *exactly which policy is running in cluster X right now* answered by a single field.

shell

# Build a signed bundle from the policies/ directory
opa build -b policies/ -o bundle.tar.gz \
  -r "v1.4.0+3f9c2d1" \
  --signing-key /keys/bundle-private.pem
 
# Verify what you just built
opa inspect bundle.tar.gz
# -> MANIFEST:
# -> +----------+----------------+
# -> |  FIELD   |     VALUE      |
# -> +----------+----------------+
# -> | Revision | v1.4.0+3f9c2d1 |
# -> | Roots    | kubernetes     |
# -> +----------+----------------+
# -> NAMESPACES:
# -> +---------------------------+-------------------------------+
# -> |         NAMESPACE         |             FILE              |
# -> +---------------------------+-------------------------------+
# -> | data.kubernetes.admission | /policies/admission/deny.rego |
# -> +---------------------------+-------------------------------+

Distribution is pull-based, more like a newspaper subscription than a courier knocking on the door. Each engine polls the bundle server on a jittered interval, meaning the timing is slightly randomised so a thousand engines do not all knock at the same second. It checks the signature against a public key, then swaps the new bundle in atomically, all or nothing. No restarts. Two channels report back the other way. The status API (application programming interface, here a small endpoint every engine uses to phone home) tells you which revision each engine has actually activated. Decision logs stream out every allow and every deny along with the input that produced it. Those logs are your audit trail, and they are also the evidence you use to promote a policy to its next stage.

opa-config.yaml

services:
  bundle_registry:
    url: https://bundles.internal.example.com
    credentials:
      bearer:
        token_path: /var/run/secrets/opa/token  # never inline tokens
 
keys:
  bundle_signer:
    algorithm: RS256
    key: ${BUNDLE_PUBLIC_KEY}    # public key only; private stays in CI
 
bundles:
  admission:
    service: bundle_registry
    resource: bundles/admission/bundle.tar.gz
    polling:
      min_delay_seconds: 30      # jittered poll window
      max_delay_seconds: 120
    signing:
      keyid: bundle_signer       # reject unsigned or tampered bundles
 
status:
  service: bundle_registry       # report active revision upstream
 
decision_logs:
  service: bundle_registry
  reporting:
    min_delay_seconds: 5
    max_delay_seconds: 10

Policy promotion pipeline

1Author & review

Git PR, unit tests in CI

2Build & sign

opa build -r v1.4.0+sha

3Distribute

bundle server, engines poll

4Audit (dryrun)

drain the violation debt

5Enforce (deny)

gate: 0 violations + soak

Retiring a rule runs the same pipeline backwards: deny -> warn -> delete. The status API and decision logs feed evidence into every gate.

Staged enforcement: dryrun, warn, deny

An enforcement action is what the engine does when something breaks a rule. Gatekeeper (the OPA-based gatekeeping component that sits in front of the Kubernetes API) gives you three of them, and they line up neatly with how that speed limit got introduced. dryrun writes the violation down and does nothing else. warn lets the request through but hands the client a warning, which kubectl prints on your terminal. deny rejects the request outright. The iron rule of this whole lesson: every new policy enters production in dryrun. Gatekeeper's audit controller also rescans objects that are already sitting in the cluster, every 60 seconds by default, so dry-run shows you the real size of your violation debt, including workloads created long before anybody wrote the policy.

shell

# Stage 1: ship the constraint in dry-run
kubectl apply -f require-team-label.yaml   # spec.enforcementAction: dryrun
# -> k8srequiredlabels.constraints.gatekeeper.sh/require-team-label created
 
# The audit controller rescans existing objects (~60s); read the debt
kubectl get k8srequiredlabels require-team-label \
  -o jsonpath='{.status.totalViolations}'
# -> 137
 
# Stage 2: remediation has driven the count to zero; surface warnings
kubectl patch k8srequiredlabels require-team-label \
  --type merge -p '{"spec":{"enforcementAction":"warn"}}'
# -> k8srequiredlabels.constraints.gatekeeper.sh/require-team-label patched
 
# Stage 3: soak period passed, violations still zero -> enforce
kubectl patch k8srequiredlabels require-team-label \
  --type merge -p '{"spec":{"enforcementAction":"deny"}}'

Every step up needs a gate, and a machine should be the one holding it. Move from dryrun to warn only once every violation is either fixed or covered by an exemption with a date on it. Move from warn to deny only after totalViolations has held at zero through a soak period, which means a stretch of ordinary traffic where you watch and change nothing. A full week of real deploys. Not a quiet weekend. A CI job that reads the constraint status will run that check every single time. A person who intends to check will not.

Zero denials is not the same as zero violations

Admission control, the checkpoint that inspects objects on their way into the cluster, only looks at them when they are created or updated. Those 137 pre-existing violators keep running, untouched and uninspected. So the moment you flip to deny the cluster looks perfectly healthy. Then a node drain or an eviction recreates one of those pods, admission rejects it, and routine maintenance becomes an outage. Gate the promotion on the audit controller's count of *existing* violations, never on the absence of new denials.

Retirement, exceptions and what breaks at scale

The lifecycle does not stop at deny. Exemptions pile up. Treat every one like a parking permit: it needs a named owner and a date it stops working. Write both into metadata and fail CI the moment one goes past due, or your exceptions quietly become your real policy. Dead rules pile up too. If decision logs show a rule has not fired in 90 days, it is a repeal candidate, and you remove it by running the rollout backwards (deny to warn to delete) so you find out whether anything was leaning on it in silence. Then there are the trade-offs you bought along with this design. Polling means minutes of lag between merge and enforcement. That delay is deliberate, and it still catches out people who expect a rule to bite the second it lands. The bundle server becomes a tier-0 dependency, the kind the whole platform stops without: engines keep the last good bundle when a fetch fails, but a freshly started node has nothing at all until its first successful pull. And one giant bundle feeding hundreds of clusters costs activation time and memory on every engine, so split it along team or domain boundaries, with roots that do not overlap, before that starts to hurt.

The stages are universal. The vocabulary is not. Revisions and status reports are OPA's spelling of the idea, enforcementAction and totalViolations are Gatekeeper's, and Kyverno (the other widely used Kubernetes policy engine) says the same things with validationFailureAction: Audit and PolicyException resources. The next lesson sets those two Kubernetes-native engines side by side: how each one handles validation, how each handles mutation, how each handles this lifecycle, and how to pick.

Try this

Run these in a lab or a throwaway cluster. You want to see the real shape of the output on your own screen, not a screenshot from somebody's blog.

terminal

git log --oneline policies/ | head
kubectl get constrainttemplates 2>/dev/null | head

output

a1b2c3d tighten image registry
b2c3d4e exception expiry for batch

Takeaway

Treat policy like a product. Review it, version it, roll it out in waves, put a name against it, and retire it when it stops earning its place. A rule with no lifecycle becomes folklore with teeth.

Next: exceptions that expire, and break-glass access (the emergency override you keep for the bad night) that closes itself behind you.

Kyverno mutate, generate, and…Exceptions with expiry

What OPA is: decoupled policyOPA & Rego What IaC scanning findsCheckov & IaC scanning Frameworks demystifiedCompliance as code Policy-as-code interview questionsInterview prep