Deny-by-default patterns that scale

Partial sets, helpers, and reusable libraries.

A hotel keycard opens one room. Every lock's resting state is shut, and nobody maintains a list of the rooms each guest may *not* enter, because that list would need rewriting every time the hotel adds a floor. Deny-by-default is the same shape in policy code: the answer is no until a specific rule says yes, and the rules that say yes are short lists you maintain on purpose. Privileged pods, public load balancers, unapproved image registries, missing ownership labels. For each one, write down what good looks like and refuse everything else.

Run it the other way and you get a catalogue of forbidden things that is permanently one release behind. You block privileged: true, so someone ships hostPID: true instead and reads every process on the node. You block that, so they mount the container runtime socket at /var/run/docker.sock and start containers outside your view. Every gap in a denylist is a running workload holding more power over the node than you meant to hand out. Allowlists have gaps too, but theirs fail in the safe direction: something legitimate gets blocked, an engineer files a ticket in ten minutes, and you fix it in daylight instead of in an incident review.

Deny-By-Default Has Two Halves

The phrase covers two separate decisions, and most teams get the first right while quietly getting the second wrong. The first is about fields: prefer a four-line allowlist of approved registries over a hundred-line denylist of banned ones, and do the same for ingress classes and for Linux capabilities. The second is about the decision itself, meaning what your policy returns when nothing matches or a field is missing. OPA (Open Policy Agent, the general-purpose policy engine most Kubernetes admission tooling is built on) evaluates a language called Rego, and in Rego a rule that does not match is not false. It is undefined, a third state that means "no opinion". Whatever called your policy decides what to do with no opinion, and the convenient answer is to admit the request.

So an entry point built only from a deny set is default-allow wearing a security costume. An empty set of violations and a policy that never loaded look identical to the caller: both are zero messages. Make the final answer an explicit allow rule with a default, and let the collected violations feed it.

package kubernetes.admission
 
import data.config
import data.lib.pod
import data.lib.registry
 
# No rule matched, a field was missing, a helper went undefined:
# every one of those roads ends here, at false.
default allow := false
 
allow if count(deny) == 0
 
deny contains msg if {
	some c in pod.all_containers(input.request.object)
	pod.is_privileged(c)
	msg := sprintf("SEC-POD-001: container %q sets securityContext.privileged=true", [c.name])
}
 
deny contains msg if {
	some c in pod.all_containers(input.request.object)
	registry.unapproved_image(c.image, config.registries)
	msg := sprintf(
		"SEC-IMG-001: container %q image %q is not from an approved registry; allowed prefixes: %s",
		[c.name, c.image, concat(", ", config.registries)],
	)
}

Two lines carry the weight. default allow := false means that if the allow body never succeeds, the value is false rather than undefined, so a caller checking allow == true gets a real refusal instead of silence. allow if count(deny) == 0 means the only route to true is a provably empty violation set.

opa eval -f pretty -I -d policy 'data.kubernetes.admission.allow' < fixtures/bad-pod.json

false

# same query, one letter dropped from the rule name
opa eval -f pretty -I -d policy 'data.kubernetes.admission.alow' < fixtures/bad-pod.json
echo "exit=$?"

undefined
exit=0

No error, no warning, and an exit code that says everything is fine. A typo in the path your webhook queries turns the strictest policy in the company into a component with no opinion about anything. Pin the decision path in one place, add --fail to the command so an undefined result exits non-zero, and smoke-test that a known-bad fixture returns false from that exact query string. Then a rename breaks the build instead of breaking production quietly.

Partial Sets Collect Every Violation

A restaurant bill that reads "there was a problem with your order" is useless. An itemized bill tells you which three dishes were wrong. Rego gives you both shapes, and the difference matters once you have more than two rules. deny := msg if { ... } is a complete rule: it holds one value. deny contains msg if { ... } is a partial set: every rule body that matches contributes one element, and the result is the union of all of them. Write ten separate deny contains blocks in a package and they all pour into the same set.

The complete-rule version does not quietly pick a winner when two bodies match with different strings. It fails the whole evaluation.

cat > /tmp/complete.rego <<'EOF'
package bad
 
deny := msg if {
	input.request.object.spec.containers[0].securityContext.privileged
	msg := "privileged container"
}
 
deny := msg if {
	startswith(input.request.object.spec.containers[0].image, "docker.io/")
	msg := "unapproved registry"
}
EOF
 
opa eval -f pretty -I -d /tmp/complete.rego 'data.bad.deny' < fixtures/bad-pod.json

1 error occurred: /tmp/complete.rego:3: eval_conflict_error: complete rules must not produce multiple outputs

The partial set has a second benefit that shows up in developer behaviour rather than in test results. A policy reporting one problem per attempt teaches people to fix, push, wait four minutes, get denied again, and repeat. Five round trips later they are hunting for the annotation that turns your controller off. Sets also remove duplicate strings, which is why every message here carries the container name: without it, three broken containers collapse into a single line.

Write The Check Once

A kitchen keeps one recipe card for the house dressing rather than trusting twenty cooks to each remember the ratio. Copy-pasting startswith(c.image, "reg.internal.example.com/") into twenty Gatekeeper templates is the twenty-cooks approach, and drift arrives on schedule. Six months later, three of those templates still allow docker.io, and nobody can tell you which three without reading all twenty. Put the check in a shared package and import it.

package lib.pod
 
# Two definitions of one function act as a logical OR: either body
# succeeding makes the call true.
is_privileged(container) if {
	container.securityContext.privileged == true
}
 
is_privileged(container) if {
	some cap in container.securityContext.capabilities.add
	cap in {"SYS_ADMIN", "SYS_PTRACE", "NET_ADMIN"}
}
 
# initContainers run first and run as root far more often than app
# containers. A rule looping over spec.containers alone never sees them.
all_containers(obj) := array.concat(
	object.get(pod_spec(obj), "containers", []),
	object.get(pod_spec(obj), "initContainers", []),
)
 
pod_spec(obj) := obj.spec if obj.kind == "Pod"
 
pod_spec(obj) := obj.spec.template.spec if {
	obj.kind in {"Deployment", "StatefulSet", "DaemonSet", "Job"}
}

package lib.registry
 
# The allowlist is the only thing that can produce a pass. Anything the
# list does not cover is unapproved by construction, including an image
# with no registry prefix at all.
unapproved_image(image, approved) if not approved_image(image, approved)
 
approved_image(image, approved) if {
	some prefix in approved
	startswith(image, prefix)
}

package config
 
# Configuration lives apart from logic, and helpers take the list as an
# argument, so a test can pass its own list without mocking anything:
#   registry.unapproved_image("docker.io/nginx", ["test.local/"])
registries := [
	"reg.internal.example.com/",
	"ghcr.io/acme-platform/",
]

all_containers is the highest-value helper in that file, because forgetting initContainers is one of the most common real bypasses in Kubernetes policy. An initContainer runs to completion before the app container starts, with the same access to the node, and a rule iterating only spec.containers admits a privileged one without comment. Centralizing container selection means every rule you write inherits the fix. pod_spec does the same for wrappers: a Deployment nests its pod under spec.template.spec, so a policy written against bare Pods sees nothing when a team deploys the normal way. Notice what happens for a kind neither definition covers. pod_spec is undefined, all_containers is undefined, and the rule produces no violations at all, which is why the list of kinds in that function has to match the list of kinds your webhook is registered for.

echo '{"request":{"object":{"kind":"Pod","spec":{"containers":[{"name":"x","securityContext":{"privileged":true}}]}}}}' \
  | opa eval -f pretty -I -d policy 'count(data.kubernetes.admission.deny)'

1

One violation, not two, and the reason is the lesson inside the lesson. That container has no image field. c.image is undefined, so passing it into registry.unapproved_image makes the whole expression undefined, the body fails, and no registry violation appears. Here it is harmless, because the Kubernetes API server rejects an imageless container on its own. On a custom resource, or on a field your policy assumed was always present, the same mechanic is a silent allow. Feed your helpers deliberately broken fixtures: a string where you expected an object, a missing key, a null.

One Library, Two Front Doors

The helpers are shared, but the entry rules cannot be, because each consumer hands you a different shape. An admission webhook receives an AdmissionReview (the JSON envelope the Kubernetes API server posts to a webhook when someone creates or updates an object), so the workload sits at input.request.object. Gatekeeper rewraps it and puts the workload at input.review.object. conftest, which runs the same policies over YAML files (the indented text format Kubernetes manifests are written in) during CI (continuous integration, the automated checks that run on every merge request), gives you the manifest as input directly. Keep entry rules thin and shape-specific, and keep every ounce of judgement in lib.

# Same helpers, different doorway: conftest hands you the manifest as
# `input`, with no AdmissionReview wrapper around it.
package main
 
import data.config
import data.lib.pod
import data.lib.registry
 
deny contains msg if {
	some c in pod.all_containers(input)
	registry.unapproved_image(c.image, config.registries)
	msg := sprintf(
		"SEC-IMG-001: container %q image %q is not from an approved registry; allowed prefixes: %s",
		[c.name, c.image, concat(", ", config.registries)],
	)
}

conftest test --policy policy --namespace main manifests/web-deploy.yaml
echo "exit=$?"

FAIL - manifests/web-deploy.yaml - main - SEC-IMG-001: container "app" image "docker.io/library/nginx:1.27" is not from an approved registry; allowed prefixes: reg.internal.example.com/, ghcr.io/acme-platform/
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
exit=1

Build the message with concat rather than dropping the list into %v and hoping. A Rego set or array formatted with %v comes out in Go's own notation, unquoted and space-separated, which is the kind of detail that turns a helpful denial into a puzzle. Gatekeeper consumes the same library without a copy-paste, because a ConstraintTemplate (the object that defines a new policy type, backed by a CRD, a custom resource definition, which is how you add your own object kinds to the Kubernetes API) accepts a libs list alongside its main rego block. The one hard rule: every library module must sit in the lib package namespace, which is why the files above are package lib.pod and package lib.registry. Generate the template from the repo at build time so the embedded strings are never hand-edited, and keep the YAML out of the directory OPA loads, or opa eval -d policy will pull your Kubernetes manifests in as data.

apiVersion: templates.gatekeeper.sh/v1
kind: ConstraintTemplate
metadata:
  name: k8snoprivileged
spec:
  crd:
    spec:
      names:
        kind: K8sNoPrivileged
  targets:
    - target: admission.k8s.gatekeeper.sh
      rego: |
        package k8snoprivileged
        import data.lib.pod
 
        violation[{"msg": msg}] {
          c := pod.all_containers(input.review.object)[_]
          pod.is_privileged(c)
          msg := sprintf("SEC-POD-001: container %q sets securityContext.privileged=true; allowed: false or unset. https://go.example.com/std/pod-security", [c.name])
        }
      # Rendered from policy/lib/*.rego by CI. Never hand-edited.
      # Library modules MUST live under the `lib` package.
      libs:
        - |
          package lib.pod
          is_privileged(container) { container.securityContext.privileged == true }
          # ...remainder generated from policy/lib/pod.rego

One caveat that costs people a day: Gatekeeper embeds its own build of OPA, so which Rego dialect it accepts depends on which Gatekeeper release you are running. Older releases parse only the pre-1.0 dialect, where rules read violation[{"msg": msg}] { ... } with no if and no contains, exactly as written above. Newer ones accept the modern syntax your conftest policies use. Check the version running in the cluster before you commit to a shared library, pin your formatter and linter to the same dialect, and render the template in CI so the two never drift apart. Getting this wrong means you own two libraries again without noticing.

Catch The Typo That Allows Everything

The most expensive bug in this style of policy is a misspelled field path. input.request.object.spec.contaners raises nothing at compile time and nothing at run time. The rule never fires, the deny set stays empty, allow comes back true, and the privileged pod ships. Tests written by the same person on the same afternoon pass, because the fixture and the typo agree with each other. The fix is to hand OPA a description of the input so it can check your spelling: a JSON Schema, which is a machine-readable description of a document's fields and their types.

# Added in a hurry, reading the path directly instead of going through
# the helper. Look closely at "contaners".
deny contains msg if {
	some c in input.request.object.spec.contaners
	c.securityContext.runAsUser == 0
	msg := sprintf("SEC-POD-002: container %q runs as uid 0", [c.name])
}

{
  "type": "object",
  "properties": {
    "request": {
      "type": "object",
      "properties": {
        "object": {
          "type": "object",
          "properties": {
            "kind": { "type": "string" },
            "metadata": { "type": "object" },
            "spec": {
              "type": "object",
              "additionalProperties": false,
              "properties": {
                "containers": { "type": "array" },
                "initContainers": { "type": "array" }
              }
            }
          }
        }
      }
    }
  }
}

opa check --schema schemas/input.json policy/

1 error occurred: policy/kubernetes/admission.rego:24: rego_type_error: undefined ref: input.request.object.spec.contaners
	input.request.object.spec.contaners
	                          ^
	                          have: "contaners"
	                          want (one of): ["containers" "initContainers"]

"additionalProperties": false is the line doing the work. Leave it out and an unknown field is a legal field, so the typo sails through. Putting it in means you must list every field your policies actually read, which is the maintenance bill for this protection, and a hand-trimmed schema covering your twenty fields costs far less to keep current than the full Kubernetes schema. Know the limit as well: the type checker only follows paths rooted at input. Inside lib.pod, the pod object arrives as a function argument, so a typo in a helper is invisible to opa check. That is exactly the code you cover with unit tests and deliberately broken fixtures. Run opa check --schema on every merge request alongside opa test, and the bug that silently disables a control becomes a compile error a reviewer can see.

Messages Are Part Of The Control

A denial that says "policy violation" generates a Slack message to your team and eventually a request for an exemption. Put four things in every message: a stable rule identifier, the field that failed, the value you saw, and the value you accept. Add a link to the internal standard so the developer reads the reasoning without booking time with you. One readable line, no internal jargon, because the audience has never opened your policy repo.

kubectl apply -f manifests/bad-pod.yaml
echo "exit=$?"

Error from server (Forbidden): error when creating "manifests/bad-pod.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [no-privileged-containers] SEC-POD-001: container "app" sets securityContext.privileged=true; allowed: false or unset. https://go.example.com/std/pod-security
exit=1

Worth knowing exactly what happened on the wire, because people assume a denial is an error and it is not. The webhook answered with a normal HTTP 200 success response (HTTP being the ordinary web protocol the API server uses to call it) carrying an AdmissionReview body whose response.allowed field is false, plus a status object holding your message. A failed admission is a successful call containing a refusal. The API server is what renders that refusal to the client as Forbidden, prefixed with the constraint name in square brackets. Anything you did not put in msg is gone by the time a human reads it.

Stable identifiers like SEC-POD-001 do work beyond readability. Dashboards group by identifier instead of message text, so rewording a message does not reset your trend line, and exceptions reference the identifier rather than a fragile substring. Where the engine supports a structured denial object rather than a bare string, use it: Gatekeeper's violation[{"msg": msg, "details": {...}}] attaches the offending field and value as data, which turns a log-scraping exercise into a query.

Audit First, Enforce On A Date

A new speed camera mails warnings for a month before it issues fines, and the point of that month is finding out how many people it will catch. Turning a new deny rule straight on across a live cluster is how a policy team loses its enforcement authority by lunchtime. Gatekeeper puts three values behind one field, spec.enforcementAction on a Constraint. dryrun blocks nothing and records violations. warn admits the request and returns a Kubernetes API warning that kubectl prints to whoever ran the command. deny, the default, returns the refusal above. Start at dryrun, read the count, then move.

kubectl get k8snoprivileged no-privileged-containers \
  -o jsonpath='{.status.totalViolations}'

37

That number comes from Gatekeeper's audit loop, which rescans objects already in the cluster on an interval (60 seconds by default) and writes results to the constraint's status. Audit runs no matter which enforcement action you set, so dryrun still fills in the number. It answers the question you need before enforcing: how much existing debt this rule will block on the next redeploy. Read status.totalViolations for the count and status.violations for examples, and remember the second one is truncated (twenty entries by default) so you can see the shape of the problem without a status object the size of a phone book. Thirty-seven is a conversation with three teams and a two-week runway. Zero means you can flip to deny today, and it also means you should confirm the rule fires at all against a deliberately bad fixture.

What This Costs

Every deny rule loops over containers, and evaluation cost is real. A comprehension nested inside that loop is quadratic, so a crafted Pod carrying hundreds of containers can turn a careless rule into seconds of processor time. Gatekeeper ships its validating webhook with a three-second timeout and a failurePolicy of Ignore, which means a slow policy is both an availability problem and a bypass: exceed the timeout and the request is admitted with no decision at all. Put the cheapest discriminator first in every rule body, benchmark a pathological input on purpose rather than finding the limit during an attack, and treat the move to failurePolicy: Fail as its own project with its own rollback plan.

Your lib package is now a shared dependency with real consumers. Renaming is_privileged breaks every template importing it, on the next bundle push, in every cluster at once. Version the bundle, use a major bump for any breaking change, write it in a changelog the app teams can find, and give people a deprecation window instead of a surprise. Review a helper change the way you would review a change to a shared authentication library, because that is the blast radius.

The honest trade-off is toil. Allowlists create an approval queue: someone has to add each new registry, each new ingress class, each new capability, and that queue is a standing operational load with a name attached to it. When nobody owns it, requests pile up, teams get blocked on a Friday afternoon, and somebody senior grants a blanket exemption worse than the denylist you replaced. Staff the queue before you flip enforcement on. A control with two-day approval latency is a control people design around.

Try This

Take a registry check that has been copy-pasted into two or more templates, move it into policy/lib/registry.rego, and prove both directions with fixtures: one Pod that must be denied, one that must pass, plus a fixture with an empty image string and one with an initContainer.

opa test ./policy -v

data.lib.pod_test.test_privileged_flag_is_detected: PASS (241.7µs)
data.lib.pod_test.test_sys_admin_capability_is_detected: PASS (198.3µs)
data.lib.pod_test.test_init_containers_are_included: PASS (312.5µs)
data.lib.registry_test.test_approved_prefix_passes: PASS (144.9µs)
data.lib.registry_test.test_empty_image_string_is_unapproved: PASS (151.2µs)
data.kubernetes.admission_test.test_clean_pod_is_allowed: PASS (402.8µs)
--------------------------------------------------------------------------------
PASS: 6/6

Everything above assumed you are writing Rego and choosing where it runs. That assumption is the next decision. Gatekeeper keeps this library-and-opa test workflow but wraps it in Kubernetes custom resources, while Kyverno drops Rego for YAML rules, trading the shared-helper model for policies your app teams can read without learning a new language. Which cost you would rather pay is the next lesson.

Takeaway

The trap worth remembering here: an empty deny set and a broken policy look identical. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related