CI parity with Conftest and opa eval

Catch violations before the cluster sees them.

Airline check-in desks keep a metal cage beside them so you can test whether your carry-on bag fits. The cage at the desk and the cage at the boarding gate are supposed to be the same size. When they are not, the desk check is worse than having no check at all: it hands people a pass they have not earned, and the real rejection happens at the gate, in front of a queue, with no time left to fix anything. CI parity is the work of making the policy cage in your pipeline the same size as the one at the cluster door. *CI* is continuous integration, the automated checks that run on every code change before it merges.

Admission control is the last door. It is the only place that can stop an object from ever existing, so it stays switched on no matter what else you build. But if admission is the only door, every policy failure surfaces at deploy time, which is the most expensive moment to learn anything. Parity does not mean running the same binary in both places, and it does not mean the pipeline replaces the cluster. It means both places give the same answer to the same question, so that a green pull request is a promise you can keep.

The Two Ways Parity Breaks

The loud failure is the one everyone complains about. CI says fine, the cluster says no, the deploy dies at 17:40 on a Friday, and somebody opens a ticket titled "policy is broken again." That is infuriating and it burns trust in the gate. It is also visible, and visible problems get fixed.

The quiet failure is the one that matters for security. Both gates go green, and neither one ever evaluated the rule. A Rego rule that reads input.spec.containers when a real Deployment nests its containers under input.spec.template.spec.containers produces zero violations, forever. It passes conftest. It passes admission. Your compliance dashboard reports full coverage across forty clusters. And a Pod with securityContext.privileged: true walks through both doors without a single alert, because nothing failed.

An attacker does not need to defeat your policy engine. They need your policy to match nothing, which is the resting state of every rule you have never run against a document shaped the way production documents are shaped. Most of this lesson is about closing that gap on purpose instead of hoping.

Conftest, the Cheapest Gate You Own

A spell-checker does not need to understand your essay. It reads the file and tells you which words are wrong, before anyone else reads a word. Conftest is that, for configuration. It is a command-line program (something you run in a terminal, with no server involved) that evaluates Rego against files sitting on disk: YAML and JSON, the two text formats Kubernetes objects are written in, plus HashiCorp Configuration Language, Dockerfiles and INI. *Rego* is the policy language that OPA evaluates, and *OPA* is the Open Policy Agent, the general-purpose decision engine from the earlier lessons. No cluster, no webhook, no network call.

Conftest looks for rules named deny, violation or warn inside a package, and it accepts suffixed variants such as deny_missing_memory_limit so each rule can carry a readable name. By default it reads the package called main, which it refers to as the *namespace*. A deny rule returns a plain string; a violation rule returns an object with a msg key, which is the same shape Gatekeeper's ConstraintTemplates use. Remember that second shape, because it is how one rule body ends up serving both tools.

package main
 
import rego.v1
 
required_labels := {"app.kubernetes.io/name", "app.kubernetes.io/part-of"}
 
deny contains msg if {
	input.kind == "Deployment"
	some container in input.spec.template.spec.containers
	not container.resources.limits.memory
	msg := sprintf("Deployment %q: container %q must set resources.limits.memory",
		[input.metadata.name, container.name])
}
 
deny contains msg if {
	input.kind == "Deployment"
	some label in required_labels
	not input.metadata.labels[label]
	msg := sprintf("Deployment %q: missing required label %q",
		[input.metadata.name, label])
}

conftest test --policy policy/ k8s/deployment.yaml
echo "exit=$?"

FAIL - k8s/deployment.yaml - main - Deployment "api": container "api" must set resources.limits.memory
FAIL - k8s/deployment.yaml - main - Deployment "api": missing required label "app.kubernetes.io/part-of"
 
2 tests, 0 passed, 0 warnings, 2 failures, 0 exceptions
exit=1

Read that summary line, because it is the part people skip. Conftest counts one test per rule per document, so two deny rules against one Deployment give you two tests, and here both of them fired. The number moves whenever your policy set or your manifest set changes, which makes it worth watching. If a policy edit ever drops that count toward zero while the exit code stays at 0, your gate has stopped gating and the pipeline still shows a green tick.

Conftest exits 1 as soon as any deny or violation rule fires, which is all the glue a required CI job needs. Add --fail-on-warn and the scale changes: 0 for a clean run, 1 when only warn rules fired, 2 when something actually failed. Careful with that flag. Almost every CI system treats any non-zero code as a failed job, so unless your script catches the codes itself, --fail-on-warn quietly promotes every advisory rule into a blocking one.

For anything beyond a human reading a log, take the structured output. It gives you the per-file success count and every message, which is what you want for pull request annotations and for a number you can chart over time.

conftest test -p policy/ -o json k8s/deployment.yaml \
  | jq '.[0] | {filename, namespace, successes, failures: [.failures[].msg]}'

{
  "filename": "k8s/deployment.yaml",
  "namespace": "main",
  "successes": 0,
  "failures": [
    "Deployment \"api\": container \"api\" must set resources.limits.memory",
    "Deployment \"api\": missing required label \"app.kubernetes.io/part-of\""
  ]
}

Render First, or You Grade the Wrong Paper

A Helm chart is a recipe, not a meal. *Helm* is the package manager most teams use to install Kubernetes applications, and the template files it ships contain loops, conditionals and placeholders. Scanning those files teaches you almost nothing about the object the API server will receive. It is worse than useless, because a template full of {{ .Values.foo }} rarely trips a rule looking for a missing memory limit, so the run comes back clean. That is a false green, manufactured by your own pipeline. Render the chart, then grade the output.

helm template api ./charts/api --values values/prod.yaml > build/rendered.yaml
grep -c '^kind:' build/rendered.yaml
conftest test --policy policy/ build/rendered.yaml

7
FAIL - build/rendered.yaml - main - Deployment "api": container "log-shipper" must set resources.limits.memory
 
14 tests, 13 passed, 0 warnings, 1 failure, 0 exceptions

Seven objects came out of a chart whose templates directory holds three files, two rules ran against each of them, and the one violation lives in log-shipper, a sidecar container contributed by a subchart that nobody on the application team knew was there. Unrendered scanning would never have seen it. Kustomize gets the same treatment with kustomize build overlays/prod. Write the rendered result to a real file rather than piping it straight in: you get an artifact you can attach to the build for debugging, and you avoid handing Conftest a stream with no filename for it to infer a parser from. If you do pipe, pass --parser yaml so the choice is explicit.

One flag is worth knowing here. Rules that reason across files, such as "every Deployment must ship with a matching NetworkPolicy," cannot work one file at a time. --combine merges every input into a single array where each element carries a path key and a contents key, so one rule can see the whole set at once. It changes the shape of input completely, so combined rules live in their own package and get their own tests.

Two OPA Commands, Two Different Jobs

opa test and opa eval get used interchangeably in blog posts, and they answer different questions. opa test is a unit test runner for your rule library: handwritten fixtures, expected outcomes, no manifests from the real world involved. It is how you prove a rule fires when it should and stays quiet when it should not.

opa test policy/ -v

data.main.test_denies_missing_memory_limit: PASS (1.42ms)
data.main.test_denies_missing_part_of_label: PASS (612.1µs)
data.main.test_allows_compliant_deployment: PASS (498.3µs)
--------------------------------------------------------------------------------
PASS: 3/3

That third test is the one people forget to write, and it is the one that catches an over-broad rule before it blocks every deploy in the estate. Coverage is measurable too, so you can gate on it: opa test policy/ --coverage --format=json emits a report with a coverage percentage and the exact line ranges no test ever touched. Untouched lines inside a deny rule are lines that have never blocked anything, anywhere.

opa eval answers the other question: given this specific document, what does the policy say right now? That is the tool for replaying the real thing. Your webhook never sees the YAML file you wrote. The API server, the component every kubectl command talks to, wraps the object in an AdmissionReview, a JSON envelope carrying the operation, the resource kind and the object itself. Which field your rule reads depends on which engine is running it. OPA deployed as the admission webhook directly receives the whole envelope, so its rules read input.request.object and the cluster asks it for data.kubernetes.admission.deny. Gatekeeper repackages the same information, handing your rule input.review.object instead. Build the envelope your engine actually gets, then evaluate the query it actually runs.

kubectl get deploy api -o json \
  | jq '{apiVersion: "admission.k8s.io/v1", kind: "AdmissionReview",
         request: {operation: "CREATE",
                   kind: {group: "apps", version: "v1", kind: "Deployment"},
                   object: .}}' > /tmp/review.json
 
opa eval --bundle policy-bundle.tar.gz --input /tmp/review.json \
  --format pretty --fail-defined 'data.kubernetes.admission.deny'
echo "exit=$?"

[
  "Deployment \"api\": container \"api\" must set resources.limits.memory"
]
exit=1

The Shape Problem, and the Tools That Fix It

Gatekeeper wraps your Rego in a ConstraintTemplate, a custom Kubernetes object that defines both the rule and the settings it will accept. Inside that template the policy reads the object from input.review.object and its tunable settings from input.parameters, which arrive from the Constraint you apply on top. Conftest hands the same rule the bare document with neither wrapper. Copy the rule body into a second file so Conftest can run it and you own two rules that agree on the day you wrote them and drift the first time somebody edits one.

Gatekeeper's own gator command removes the guesswork. Point it at your ConstraintTemplates, your Constraints and your rendered objects, and it runs the same evaluation path the cluster runs, offline.

gator test --filename=policies/ --filename=build/rendered.yaml
echo "exit=$?"

[require-memory-limits] Deployment <api>: container <log-shipper> must set resources.limits.memory
exit=1

The exit code follows the constraint's enforcementAction, deliberately mirroring the webhook. A violation of a constraint set to deny returns 1. A violation of one set to warn or dryrun prints to standard output and leaves the exit code at 0, exactly as the cluster would admit the object while recording the finding. That is the correct design and it is also a trap: a rule you moved to dryrun during a rollout wave produces a noisy, passing CI job that nobody reads. Take gator test --output json and every violation arrives with its constraint and its enforcement action attached, so you can count how many of your rules are currently non-enforcing and put that number somewhere people look. gator verify is the other half, running Suite files of tests and cases with the expected outcome written down, which gives ConstraintTemplate authors the safety net opa test gives Rego authors.

Kyverno tells the same story in a different accent. Its policies are YAML rather than Rego, and its command-line tool applies them offline against files.

kyverno apply policies/ --resource build/rendered.yaml
echo "exit=$?"

Applying 4 policy rule(s) to 7 resource(s)...
 
policy require-resource-limits -> resource default/Deployment/api failed:
1. autogen-check-limits: validation error: memory limit is required.
   rule autogen-check-limits failed at path /spec/template/spec/containers/1/resources/limits/memory/
 
pass: 5, fail: 1, warn: 0, error: 0, skip: 1
exit=1

Look at the rule name in that output. You wrote a rule targeting Pods; the failure names autogen-check-limits. Kyverno auto-generates matching rules for the controllers that create Pods, including Deployment, StatefulSet, DaemonSet, Job and CronJob, so a policy written once against Pods also covers the objects people actually deploy. Any CI check that skips that generation step is testing a rule the cluster never uses. kyverno test runs a directory of expected-result files the same way gator verify does. This is the whole argument for reaching for the engine's own tool instead of hand-rolling an equivalent.

Mutation Makes Naive Parity Lie

At a passport desk, one clerk stamps and corrects your form before the inspector ever reads it. Judging the version in your pocket tells you nothing about what the inspector saw. Mutating webhooks work like that clerk: they rewrite an object before any validating webhook sees it. Gatekeeper Assign resources, Kyverno mutate rules and service mesh sidecar injection all do it. The cluster therefore validates a document that exists nowhere in your repository. Grade the pre-mutation file in CI and you are answering a question the API server never asks, and it goes wrong in both directions: CI can deny something mutation would have fixed, and CI can allow something mutation makes non-compliant.

A server-side dry run is the honest check. It walks the entire admission chain (authentication, then authorization, then mutation, then validation) and returns the object as the API server would have stored it, without storing it.

kubectl apply -f build/rendered.yaml --dry-run=server -o yaml \
  | yq 'select(.kind == "Deployment") | .spec.template.spec.containers[0].securityContext'

runAsNonRoot: true
seccompProfile:
  type: RuntimeDefault

Neither field appears in the rendered file. A mutation policy added both, and a CI check reading the file on disk would have called the workload non-compliant and blocked a perfectly good deploy. When something genuinely violates policy, the dry run hands back the real denial text, which is the message you want quoted into the pull request comment so a developer can act on it without opening a runbook.

kubectl apply -f build/bad.yaml --dry-run=server

Error from server (Forbidden): error when creating "build/bad.yaml": admission webhook
"validation.gatekeeper.sh" denied the request: [require-memory-limits] Deployment "api":
container "api" must set resources.limits.memory

Dry run has real costs, so be deliberate about where you spend them. It needs cluster credentials and permission to create the object, which a pull request opened from a fork must never have. It puts load on the same webhooks serving production admission. And a webhook can refuse it outright: every webhook declares a sideEffects value, and if any matching webhook says Unknown or Some, meaning it might change state somewhere outside the API server, the API server rejects the dry run rather than risk it.

kubectl apply -f build/rendered.yaml --dry-run=server

Error from server (BadRequest): admission webhook "mutate.internal.example.com" does not support dry run

That is a fixable configuration problem, not a reason to abandon the check. A webhook that truly changes nothing outside the request should declare sideEffects: None; one that avoids side effects only during dry runs declares NoneOnDryRun and gets skipped for them. Owners of internal webhooks tend to leave the field at Unknown because nothing ever forced them to think about it. Run the dry-run parity job against a staging cluster on merge or overnight, and keep the fast offline checks on every push.

Pin Both Sides to the Same Rulebook

A nickname can be quietly reassigned to a different person. A fingerprint cannot. Two gates running the same tool against different versions of the rules agree only by coincidence, so ship policy as a versioned bundle, publish it to a registry, and pin it by content digest, the SHA-256 fingerprint computed from the bytes of the artifact, rather than by a tag anyone can repoint tomorrow. A CI job pulling :latest while clusters run v1.4.2 will happily approve a deployment that relies on an exception the clusters have not received yet.

name: policy
on: [push, pull_request]
 
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
 
      # Self-tests are never skipped by path filters. A policy edit that
      # ships without tests is the failure mode this whole job exists to stop.
      - name: Policy self-tests
        run: |
          opa test policy/ --coverage --format=json > coverage.json
          jq -e '.coverage >= 80' coverage.json
 
      - name: Render what we will actually apply
        run: |
          mkdir -p build
          helm template api ./charts/api -f values/prod.yaml > build/rendered.yaml
 
      # --output github emits workflow annotations, so each deny message
      # lands on the offending line of the diff instead of inside a log blob.
      - name: Gate
        run: conftest test --policy policy/ --output github build/rendered.yaml
 
      # Publish the exact rules this commit produced and print the digest.
      # The GitOps repo (the git repository the clusters sync from) pins
      # every cluster to that digest, so both sides run one rulebook.
      - name: Publish the bundle
        if: github.ref == 'refs/heads/main'
        run: |
          opa build --bundle policy/ --output bundle.tar.gz
          oras push ghcr.io/acme/policy-bundle:${{ github.sha }} bundle.tar.gz

Caching the bundle keeps pull request feedback quick, and a cache that silently serves yesterday's rules is a gate that stopped working without telling anyone, so verify the digest after every restore. Because OPA records the bundle revision in its decision logs, an auditor asking "which rules allowed this?" six months from now gets an exact answer instead of a shrug.

Prove the Gate Is Alive

Smoke alarms get tested with actual smoke, not by looking at the green light. Keep a fixture that violates a rule nobody could ever have a legitimate reason to relax, then assert that both gates reject it. Run it on a schedule, once per cluster.

# fixtures/known-bad.yaml: privileged: true, no limits, no required labels.
conftest test -p policy/ fixtures/known-bad.yaml >/dev/null 2>&1 \
  && echo "CI-ALLOWED" || echo "CI-DENIED"
kubectl apply -f fixtures/known-bad.yaml --dry-run=server >/dev/null 2>&1 \
  && echo "K8S-ALLOWED" || echo "K8S-DENIED"

CI-DENIED
K8S-DENIED

Two matching denials mean the control works today. The day you see CI-ALLOWED above K8S-DENIED, your pipeline has been lying to developers for however long that drift has existed, and every green tick since then meant nothing. The reverse pairing, CI-DENIED over K8S-ALLOWED, points somewhere more urgent: something in the cluster is not enforcing, whether that is a constraint left on dryrun after a rollout, a namespace exclusion somebody added during an incident, or a webhook falling back to its Ignore failure policy. Compare the two results as a time series and treat divergence as a page-worthy signal, not a chart nobody opens.

Try This

Run this in a scratch repository so you see the real exit codes rather than a screenshot of them. Write one Deployment that violates the memory-limit rule and one that satisfies it, then confirm the gate tells them apart.

conftest test --policy policy/ fixtures/bad.yaml || echo "blocked (exit $?)"
conftest test --policy policy/ fixtures/good.yaml && echo "allowed (exit $?)"
opa test policy/ --coverage --format=json | jq '.coverage'

FAIL - fixtures/bad.yaml - main - Deployment "bad": container "app" must set resources.limits.memory
 
2 tests, 1 passed, 0 warnings, 1 failure, 0 exceptions
blocked (exit 1)
 
2 tests, 2 passed, 0 warnings, 0 failures, 0 exceptions
allowed (exit 0)
86.4

The honest trade-off is worth stating plainly. Every check described here costs pipeline minutes and somebody's ongoing attention, and none of them removes the need for admission control, because anyone holding cluster credentials can walk past your pipeline entirely. What they buy is a shorter feedback loop and, more usefully, evidence: a failing canary tells you the two gates have drifted apart before an auditor or an incident does. A CI gate you never test is decoration that makes the deploy slower.

Next: distributing those pinned bundles across a fleet of clusters without ending up with a snowflake configuration per environment.

Takeaway

The trap worth remembering here: --fail and --fail-defined are opposites, and picking the wrong one fails open. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related