Documents, data, and packages

input, data, and how Rego loads the world.

A policy engine behaves more like a librarian than a robot. You hand it a document, it checks that document against the rules it has memorized and the reference binders on its shelf, and it hands back an answer. Rego, the language that OPA (Open Policy Agent, a general-purpose engine for making yes-or-no decisions about structured data) is programmed in, never logs into your cluster, never calls an API (application programming interface, the endpoint one program uses to talk to another), never runs a script against anything. It answers questions about JSON (JavaScript Object Notation, the nested key-and-value format Kubernetes speaks) trees that somebody else placed in front of it.

Two of those trees carry all the weight. input is the document under review right now. data is everything loaded ahead of time: approved registry prefixes, which team owns which namespace, the exception list and its expiry dates. Packages decide where your rules live inside data, so data.kubernetes.admission.deny and data.terraform.deny never trip over each other. Nearly every baffling Rego bug traces back to one of those three being different in production from what it was on your laptop.

The Form on the Desk and the Binder on the Shelf

A passport officer works from two things: the form you hand over, and the reference binder beside the counter. The form changes every thirty seconds. The binder changes when policy changes. input is the form. data is the binder. A rule that reads the binder as though it were the form, or the other way round, will pass every test you write and then behave like a stranger in production, because in your test you stuffed both into the same object.

mkdir -p /tmp/pol && cat > /tmp/pol/registry.rego <<'EOF'
package example
 
deny contains msg if {
  input.kind == "Pod"
  some c in input.spec.containers
  not startswith(c.image, data.allowed.registry)
  msg := sprintf("image registry not allowed: %v", [c.image])
}
EOF
 
echo '{"allowed":{"registry":"ghcr.io/acme/"}}' > /tmp/pol/data.json
 
echo '{"kind":"Pod","spec":{"containers":[{"image":"docker.io/library/nginx:1.25"}]}}' \
  | opa eval -f pretty -I -d /tmp/pol 'data.example.deny'

[
  "image registry not allowed: docker.io/library/nginx:1.25"
]

Read the flags, because they are the whole lesson in miniature. -I tells opa eval to take standard input and mount it as input. -d tells it to load these files into data. Two separate doors into the engine. The result prints as a JSON array because deny contains msg builds a set, and sets come out as arrays once they are written to JSON. The syntax here (if, contains, some ... in) is OPA 1.x. On an older 0.x binary the same file needs import rego.v1 at the top, and 1.x still accepts that line and ignores it, so writing it costs you nothing and buys you portability.

Where Data Actually Lands

People assume the filename decides the path. It does not. OPA builds the data path out of directory names and throws the file's own name away. A file called registries.json sitting at the root of your load directory merges straight into the top of data. Move it one folder down into registries/ and every key inside it shifts to data.registries.*. Nothing warns you and nothing fails to compile. Query the whole document when you want the truth.

mkdir -p /tmp/pol/registries && mv /tmp/pol/data.json /tmp/pol/registries/data.json
opa eval -f pretty -d /tmp/pol 'data'

{
  "example": {
    "deny": []
  },
  "registries": {
    "allowed": {
      "registry": "ghcr.io/acme/"
    }
  }
}

Two things to notice. The registry list moved to data.registries.allowed.registry, so data.allowed.registry is now nothing at all. And your rules show up in the same tree as the facts: data.example.deny evaluated to an empty set because there was no input this time. Rules and facts share one namespace, which is exactly why packages exist. Now run the policy again against an image that should be perfectly fine.

echo '{"kind":"Pod","spec":{"containers":[{"image":"ghcr.io/acme/api:1.4"}]}}' \
  | opa eval -f pretty -I -d /tmp/pol 'data.example.deny'

[
  "image registry not allowed: ghcr.io/acme/api:1.4"
]

An approved image, denied. Nobody edited the rule. Somebody tidied a folder. startswith(c.image, data.allowed.registry) could not resolve its second argument, so the whole expression became undefined, and in Rego not undefined succeeds. The rule went from "block unapproved registries" to "block everything" in one git mv. That is a self-inflicted outage across every namespace the constraint matches, and it lands at deploy time on a Friday afternoon.

One Policy, Three Input Shapes

The same question ("is this image from an approved registry?") gets asked by different tools, and each one hands you a differently wrapped envelope. Gatekeeper nests the object under input.review.object and puts tunable values under input.parameters. A plain OPA validating webhook receives the raw AdmissionReview that the API server sends and sees input.request.object. Conftest, running in CI (continuous integration, the automated checks that gate a merge), hands you the parsed file itself at the top level. Feed the wrong envelope to a working rule and watch what comes back.

mv /tmp/pol/registries/data.json /tmp/pol/data.json && rmdir /tmp/pol/registries
 
echo '{"review":{"object":{"kind":"Pod","spec":{"containers":[{"image":"docker.io/library/nginx:1.25"}]}}}}' \
  | opa eval -f pretty -I -d /tmp/pol 'data.example.deny'

[]

Empty set. No error, no warning, exit code 0. input.kind does not exist in a Gatekeeper-shaped document, the first line of the body failed, and the rule produced nothing. An admission controller (the part of the Kubernetes API server that gets to inspect a write before it is stored) reads "nothing" as "admit it". This is how a team ships a registry allowlist, sees green unit tests, applies the ConstraintTemplate, and runs unprotected for four months until somebody pulls a cryptominer image from a public registry and it sails in. The same trap has two smaller cousins worth knowing. A rule that reads spec.containers never looks at initContainers or ephemeralContainers, so an unapproved image hidden in an init container walks straight past. And a rule written for a Pod finds nothing at all in a Deployment file, where the containers sit at spec.template.spec.containers, which matters the moment you point conftest at a repository full of Deployments.

Packages Are Addresses, Entry Points Are Contracts

A package name is a street address inside data, and it has nothing to do with where the file sits on disk. Ten files can declare package lib.images and their rules merge into one place. The name you give the top rule is not decoration either. It is a contract with whatever calls you. Gatekeeper looks for violation and expects every element to be an object carrying a msg key. Conftest looks for deny, violation or warn inside whichever namespace you point it at. Rename the entry point and your policy still compiles, still passes its own tests, and enforces nothing whatsoever.

The envelope differs per tool. The security question does not. So put the real logic in a helper that takes the object as an argument, then write a two-line adapter for each place you enforce.

package lib.images
 
# offending returns the set of images in obj that match no allowed prefix.
# Checks init and ephemeral containers too, because attackers read specs.
offending(obj, prefixes) := {img |
  some c in all_containers(obj)
  img := c.image
  not any_prefix(img, prefixes)
}
 
all_containers(obj) := array.concat(
  array.concat(
    object.get(obj, ["spec", "containers"], []),
    object.get(obj, ["spec", "initContainers"], []),
  ),
  object.get(obj, ["spec", "ephemeralContainers"], []),
)
 
any_prefix(img, prefixes) if {
  some p in prefixes
  startswith(img, p)
}

package conftest.images
 
import data.lib.images
 
deny contains msg if {
  input.kind == "Pod"
  some img in images.offending(input, data.registries.prefixes)
  msg := sprintf("image %v is not from an approved registry", [img])
}

{
  "prefixes": [
    "ghcr.io/acme/",
    "registry.acme.internal/"
  ]
}

Look at where that fact file sits before you run anything. The rule asks for data.registries.prefixes, and the only reason that path exists is the folder named registries. The filename data.json contributes nothing. Put the same content in facts/data.json and the keys land at data.prefixes, the rule reads undefined, and you are back in the failure from two sections ago.

conftest test /tmp/pod.yaml -p policy/ -d facts/ --namespace conftest.images; echo "exit=$?"

FAIL - /tmp/pod.yaml - conftest.images - image docker.io/library/nginx:1.25 is not from an approved registry
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
exit=1

That non-zero exit code is the entire value of a CI gate. Notice the two doors again: -p loads policy, -d loads facts into data. Forget -d and data.registries.prefixes is undefined, any_prefix can never succeed, every image drops into the offending set, and the pipeline goes red for the whole organization. Somebody appends || true to the step by lunchtime, and the control is dead with everyone's approval. One more version note: conftest carries its own copy of OPA inside it, so a build older than roughly v0.56 will reject the if and contains keywords unless you add import rego.v1. Run conftest --version before you blame your rule.

Audit, Enforce, and What a Denial Actually Returns

Gatekeeper splits one policy into two objects. The ConstraintTemplate carries the Rego and generates a CRD (custom resource definition, which is how you teach Kubernetes a brand new object type). The Constraint is an instance of that new type: it says which resources to match and supplies the parameters. One template, many constraints, different values per environment.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8sallowedregistries
spec:
  crd:
    spec:
      names:
        kind: K8sAllowedRegistries
      validation:
        openAPIV3Schema:
          type: object
          properties:
            prefixes:
              type: array
              items:
                type: string
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8sallowedregistries
 
        import rego.v1
 
        violation contains {"msg": msg} if {
          some c in input.review.object.spec.containers
          not allowed(c.image)
          msg := sprintf("image %v is not from an approved registry (allowed prefixes: %v)", [c.image, concat(", ", input.parameters.prefixes)])
        }
 
        allowed(image) if {
          some p in input.parameters.prefixes
          startswith(image, p)
        }

That rego: block is compiled by the copy of OPA baked into the Gatekeeper image, which trails the standalone binary you have on your laptop. Gatekeeper 3.15 and newer accept import rego.v1, so the if and contains keywords work. Older releases need the pre-1.0 shape, violation[{"msg": msg}] { ... }, with no if anywhere. Check what the cluster is actually running with kubectl -n gatekeeper-system get deploy gatekeeper-controller-manager -o jsonpath='{.spec.template.spec.containers[0].image}' before you paste a template in and wonder why it will not compile.

apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRegistries
metadata:
  name: prod-registries
spec:
  enforcementAction: dryrun
  match:
    kinds:
      - apiGroups: [""]
        kinds: ["Pod"]
    namespaces: ["payments"]
  parameters:
    prefixes:
      - "ghcr.io/acme/"
      - "registry.acme.internal/"

enforcementAction: dryrun means admission always says yes. Nothing is blocked, nothing comes back to the person running kubectl, and the only trace is the audit loop writing findings onto the Constraint's status roughly every sixty seconds. Start here every single time, because audit hands you the blast radius before you own the pager for it.

kubectl get k8sallowedregistries prod-registries -o json \
  | jq '{total: .status.totalViolations, listed: (.status.violations | length), first: .status.violations[0]}'

{
  "total": 41,
  "listed": 20,
  "first": {
    "enforcementAction": "dryrun",
    "group": "",
    "kind": "Pod",
    "message": "image docker.io/library/nginx:1.25 is not from an approved registry (allowed prefixes: ghcr.io/acme/, registry.acme.internal/)",
    "name": "checkout-7c9f8b6d4-2xk9p",
    "namespace": "payments",
    "version": "v1"
  }
}

Forty-one workloads would break if you flipped this today. That number is your migration plan. The listed count stops at 20 because Gatekeeper caps how many violations it writes into the status object, controlled by --constraint-violations-limit and set to 20 by default, so the resource does not grow without limit. Trust totalViolations for the size of the problem and the audit pod's logs for the full list. Fix or exempt the offenders, then change one field and try the same Pod again.

kubectl patch k8sallowedregistries prod-registries --type merge \
  -p '{"spec":{"enforcementAction":"deny"}}'
kubectl -n payments apply -f /tmp/pod.yaml

k8sallowedregistries.constraints.gatekeeper.sh/prod-registries patched
Error from server (Forbidden): error when creating "/tmp/pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [prod-registries] image docker.io/library/nginx:1.25 is not from an approved registry (allowed prefixes: ghcr.io/acme/, registry.acme.internal/)

Look closely at what came back, because people describe this wrongly in incident reviews. It is an HTTP 403 from the API server, printed by kubectl as Forbidden. Gatekeeper's webhook (a small HTTPS service the API server calls out to before storing an object) returned an admission response with allowed: false plus a status message, and Gatekeeper stamped the name of the Constraint that produced it in front of that message. The bracketed [prod-registries] is the best debugging gift in the system: it tells the developer exactly which object to go and read. Nothing was deleted and nothing was rolled back, because the write never happened. There is a middle setting too. enforcementAction: warn admits the object and returns the same sentence as a client-side warning, which is how you let people feel a policy before it bites them. Pods already running are untouched by all three settings, because admission only ever inspects writes to the API. Those 41 offenders keep running happily until something recreates them, and that is the moment they stop.

Pulling Cluster State Into data

Some questions need a second document on the desk. "Does this Ingress hostname clash with one in another namespace?" cannot be answered from the object under review by itself. Gatekeeper can hold a cached copy of chosen kinds and expose them at data.inventory.namespace[ns][groupVersion][kind][name] for namespaced objects, and data.inventory.cluster[groupVersion][kind][name] for cluster-scoped ones. You opt in one kind at a time with a Config resource.

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config
  namespace: gatekeeper-system
spec:
  sync:
    syncOnly:
      - group: ""
        version: "v1"
        kind: "Namespace"
      - group: "networking.k8s.io"
        version: "v1"
        kind: "Ingress"

Bundles Pin the Whole World

Rego plus the data files it depends on is a single artifact, and OPA can package it the way you package a container image. opa build produces a gzipped tar file with a manifest inside that records a revision string. Point OPA at a bundle server, let it poll, and every decision the cluster makes is traceable to one version of both the rules and the facts.

opa build -o bundle.tar.gz -r "$(git rev-parse --short HEAD)" policy/ facts/
opa inspect -f json bundle.tar.gz | jq '.manifest'

{
  "revision": "9f31c0a",
  "roots": [
    ""
  ]
}

Note that there is no -b on that build. --bundle is an on/off flag meaning "the paths I am giving you are already bundles", and policy/ and facts/ are ordinary directories, so passing it would be wrong. The revision is what makes "but it passed CI" an answerable claim. If the pipeline evaluated bundle 9f31c0a and the cluster is still serving 4b70de2 because its download has been failing for six days, the two disagree and both are behaving exactly as designed. Log the revision alongside every decision and alarm on staleness, which OPA will tell you about through its status plugin and its /health?bundles=true endpoint. Sign bundles with opa build --signing-key and, this is the half everyone forgets, make the OPA that consumes them verify with --verification-key or the matching signing block in its service configuration. A signature nobody checks is decoration. An unsigned, unversioned policy store is a control surface an attacker can edit far more quietly than your cluster.

Try This

Take the registry rule from the top of this lesson and give it a second life. Save one Pod fixture that must be denied and one that must pass. Run both through opa eval -I -d /tmp/pol and confirm you get one message and then []. Now move data.json into facts/data.json without touching a character of the rule, and run the good fixture again. Watch a clean Pod get denied, and sit for a moment with how invisible that failure is from the rule's source code. Then wrap both fixtures in {"review":{"object": ... }} and watch the opposite failure, where the bad Pod comes back as []. Two commands, two directions of wrong, and a solid reason to keep golden fixtures for every enforcement point you run. Next you will turn those fixtures into opa test cases so the build fails instead of production.

Takeaway

The trap worth remembering here: undefined spreads in both directions. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related