Multi-cluster distribution

GitOps, bundles, and wave rollouts.

A restaurant chain with fifty branches does not paint the food-safety rules on each kitchen wall and hope for the best. It publishes one numbered handbook, ships a dated revision to every branch, and can name the branches still working from last year's copy. Fifty Kubernetes clusters with policies applied by hand are fifty painted walls. Somebody patched a rule on the busy cluster at 2am, nobody copied it back into the shared repository, and the real policy of your estate becomes whatever each cluster half-remembers.

Multi-cluster distribution turns one reviewed rule into the same enforced rule on every cluster, carrying a version number you can point at. Three pieces do the work. GitOps (an operating model where a Git repository is the only place you change infrastructure, and an agent running inside each cluster continuously pulls that repository and makes the cluster match it) gives you one source of truth. Bundles (the gzipped tar archive of policy and data that OPA, the Open Policy Agent, downloads and swaps in while it keeps running) give you a versioned artifact you can name. Waves (shipping the new version to a few clusters, watching them, then widening) give you a blast radius you chose in advance instead of one you discovered.

Fifty Clusters, Fifty Truths

Hand-applied policy fails in two directions, and the two look nothing alike. The loud failure is blast radius. Someone tightens the rule that says every container image must come from the internal registry, forgets that the shop-floor clusters pull a vendor's agent from a public registry, and applies the change everywhere in one go. Every one of those clusters starts rejecting deployments in the same minute. No smaller group caught it first, because there was no smaller group.

The quiet failure is drift, and drift is the one that helps an attacker. A cluster whose bundle download has been failing for six weeks keeps serving the last bundle it managed to fetch, answers its health checks normally, and enforces last quarter's rules. Your dashboard says the policy merged, and it did. That cluster never received it. A cluster that was never registered with the GitOps controller in the first place is worse, because it enforces nothing and it is invisible: the list you check is built from the clusters that registered.

There is a direct attack path here too. Whatever serves policy to your engines decides what is allowed into your clusters. If fifty clusters all point at ghcr.io/acme/policy:latest, then anyone who can push that one tag has rewritten admission control on fifty clusters inside a poll interval, with no pull request, no reviewer, and no line in git log for anyone to find afterwards. Treat the distribution channel as production infrastructure with production access controls: signed artifacts, pinned versions, and a promotion path a named person approves.

One Repository, Cluster Classes, Local Parameters

Fire codes are written once for a whole country, then applied with local numbers, because a hospital needs wider corridors than a garden shed. Fleet policy works the same way. A ConstraintTemplate (Gatekeeper's reusable rule: Rego, the policy language OPA evaluates, plus a typed schema describing the settings that rule accepts) is the national code. A Constraint (one instance of that template with the settings filled in and a scope attached) is the local number. Templates stay common. Parameters go local. Copy Rego into a per-cluster directory and edit it there, and you now maintain two rules that will quietly disagree within a quarter.

Group clusters by class rather than by name. Use sandbox for the clusters engineers break on purpose, pci for the ones handling card data that an auditor will ask about, edge for the ones sitting in shops behind flaky links. A class is the unit you make a decision about: which registries are allowed, which enforcement action applies, what happens when the engine cannot be reached. A directory named after one cluster invites a fork. A directory named after a class invites a parameter.

policy-fleet/
├─ base/                             # one copy of every rule, no exceptions
│  ├─ templates/                     # ConstraintTemplates: the only place Rego lives
│  │  ├─ allowed-registries.yaml
│  │  └─ require-nonroot.yaml
│  └─ constraints/                   # constraints with placeholder parameters
│     └─ allowed-registries.yaml
├─ classes/
│  ├─ sandbox/kustomization.yaml     # dryrun, wide registry list
│  ├─ pci/kustomization.yaml         # deny, internal registry only
│  └─ edge/kustomization.yaml        # deny, plus the vendor agent registry
└─ clusters/
   ├─ canary-1/policy-version.yaml   # class: sandbox, wave: 0
   ├─ nonprod-eu-1/policy-version.yaml
   └─ prod-eu-3/policy-version.yaml  # class: pci,     wave: 2

resources:
  - ../../base                 # same templates and constraints as everyone else
 
patches:
  - target:
      kind: K8sAllowedRegistries
      name: allowed-registries
    patch: |-
      - op: replace
        path: /spec/enforcementAction
        value: deny            # sandbox keeps dryrun; pci does not
      - op: replace
        path: /spec/parameters/registries
        value: ["registry.internal.example.com/"]

Be precise about what that one word changes, because teams lose whole afternoons to it. With enforcementAction: deny, a matching request is rejected outright. With dryrun, the request is admitted, the violation is recorded in the constraint's status and counted by the audit controller, and the person deploying sees absolutely nothing. With warn, the request is also admitted, but the message is handed back to the client as a warning line above the usual output. Only deny blocks anything.

Clusters also differ in what they contain, and that changes what a rule is able to see. A referential constraint (a rule that has to look at objects other than the one being admitted, such as "no two Ingress objects may claim the same hostname") only works if Gatekeeper already holds a cached copy of those objects. It caches exactly what you list in its Config resource and nothing else. Listing every kind is how a cluster running five thousand Pods turns its policy engine into the hungriest process on the node.

apiVersion: config.gatekeeper.sh/v1alpha1
kind: Config
metadata:
  name: config                 # the name must be exactly "config"
  namespace: gatekeeper-system
spec:
  sync:
    syncOnly:                  # replicate only what referential rules read
      - group: ""
        version: "v1"
        kind: "Namespace"
      - group: "networking.k8s.io"
        version: "v1"
        kind: "Ingress"

Pin the Bundle, Never Float a Tag

Pinning is the difference between telling a courier "deliver the current handbook" and "deliver revision 2026.07.24". Build the bundle once in CI (continuous integration, the automated build and test system that runs on every merge), sign it, push it to a registry, and write down what you pushed. The revision string should answer two questions at once: which release train this belongs to, and which commit produced it.

# Build a signed bundle whose revision names the train and the commit
opa build -b policies/ -o bundle.tar.gz \
  -r "2026.07.24+3f9c2d1" \
  --signing-key /keys/bundle-private.pem
 
# Push it as an OCI (Open Container Initiative) artifact: the same registry
# format container images use, so your existing registry already stores it
oras push ghcr.io/acme/policy:2026.07.24 \
  --artifact-type application/vnd.oci.image.layer.v1.tar+gzip \
  bundle.tar.gz:application/vnd.oci.image.layer.v1.tar+gzip

✓ Uploaded  bundle.tar.gz
✓ Uploaded  application/vnd.oci.empty.v1+json
Pushed [registry] ghcr.io/acme/policy:2026.07.24
ArtifactType: application/vnd.oci.image.layer.v1.tar+gzip
Digest: sha256:6b19f8b0a2c4d1e7f0a9c3b5d8e2f4a6c9b1d3e5f7a9c2b4d6e8f0a2c4b6d8e0

A tag is a label somebody can move. A digest is a fingerprint of the content itself: change one byte of the bundle and the digest changes, so a pinned digest cannot be swapped underneath you. Put the digest in the GitOps repository, keep the tag next to it for humans to read, and tag the Git commit as well, so that a year from now an auditor can line up the artifact with the source that produced it.

git tag policy-bundle-2026.07.24
 
cat > clusters/canary-1/policy-version.yaml <<'EOF'
wave: 0
class: sandbox
bundleTag: "2026.07.24"   # readable, and moveable by anyone with push rights
bundleDigest: "sha256:6b19f8b0a2c4d1e7f0a9c3b5d8e2f4a6c9b1d3e5f7a9c2b4d6e8f0a2c4b6d8e0"
EOF
 
git add . && git commit -m 'pin policy bundle 2026.07.24 on canary-1'

[main 8d41f0c] pin policy bundle 2026.07.24 on canary-1
 1 file changed, 4 insertions(+)

services:
  ghcr:
    url: https://ghcr.io
    type: oci
    credentials:
      bearer:
        token_path: /var/run/secrets/opa/registry-token   # never inline a token
 
keys:
  bundle_signer:
    algorithm: RS256
    key: ${BUNDLE_PUBLIC_KEY}        # public half only; the private key stays in CI
 
bundles:
  admission:
    service: ghcr
    resource: ghcr.io/acme/policy@sha256:6b19f8b0a2c4d1e7f0a9c3b5d8e2f4a6c9b1d3e5f7a9c2b4d6e8f0a2c4b6d8e0
    persist: true                    # keep the last good bundle on local disk
    signing:
      keyid: bundle_signer           # refuse unsigned or altered bundles
    polling:
      min_delay_seconds: 60
      max_delay_seconds: 120

Two small things in that file carry most of the weight. signing.keyid makes OPA verify the bundle's signature before activating it, so a tampered artifact is rejected rather than enforced, and the digest in resource means the registry cannot hand you different bytes for the same reference. Note what signing does not do: the bundle is signed, not encrypted, so anyone who can read the registry can read your rules. Sign for integrity, use registry permissions for secrecy.

With that in place you can ask a cluster what it is running instead of asking Git what you merged. OPA writes the activated bundle's manifest into its own data tree, and its health endpoint can be told to report unhealthy until every configured bundle has activated at least once, which is exactly what a freshly booted node should say while it is still empty.

kubectl -n opa port-forward deploy/opa 8181:8181 >/dev/null 2>&1 &
sleep 2
 
# Which revision is actually loaded on this cluster right now?
curl -s localhost:8181/v1/data/system/bundles/admission/manifest
 
# 200 only after every configured bundle has activated at least once
curl -s -o /dev/null -w '%{http_code}\n' 'localhost:8181/health?bundles'

{"result":{"revision":"2026.07.24+3f9c2d1","roots":["kubernetes"]}}
200

Port-forward rather than kubectl exec, by the way. The official OPA image ships the binary and little else, so there is no shell and no curl inside the container to run. Loop that pair of calls across the fleet and you can answer the question that actually matters during an incident: which revision is each cluster serving at this moment. Gatekeeper has no bundles, so its equivalent marker is the Git revision its sync agent last applied, read back from the agent rather than from the repository.

Rolling in Waves

Nobody repaints every branch of the chain on the same night. A wave is a group of clusters that receives a version together and gets watched before the next group is allowed to have it. Three waves is the usual shape: one or two canary clusters carrying real but forgiving traffic, then everything non-production, then production, with the class an auditor cares about going last. The rule that makes waves worth the trouble is unglamorous. A wave finishes when the signals you named in advance stayed inside their limits for a stated soak time, not when the sync agent reports success.

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: policy-fleet
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]   # fail loudly on an unlabelled cluster
  strategy:
    type: RollingSync              # progressive syncs are alpha and off by
    rollingSync:                   # default: run the applicationset controller
      steps:                       # with --enable-progressive-syncs
        - matchExpressions:
            - { key: wave, operator: In, values: ["0"] }
        - matchExpressions:
            - { key: wave, operator: In, values: ["1"] }
        - matchExpressions:
            - { key: wave, operator: In, values: ["2"] }
  generators:
    - clusters:                    # one Application per registered cluster
        selector:
          matchLabels:
            policy-managed: "true"
  template:
    metadata:
      name: 'policy-{{.name}}'
      labels:
        policy-managed: "true"     # the steps above match these Application labels
        wave: '{{index .metadata.labels "wave"}}'
    spec:
      project: platform
      source:
        repoURL: https://github.com/acme/policy-fleet.git
        targetRevision: policy-bundle-2026.07.24          # the Git tag, pinned
        path: 'classes/{{index .metadata.labels "class"}}' # class picks the params
      destination:
        name: '{{.name}}'          # by name, so tooling can print it back
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Two details decide whether that manifest behaves like a real rollout. The first is that the controller only moves to the next step once every Application in the current step reports both Synced and Healthy, so the health of a policy Application has to mean something beyond "the YAML arrived". The second is an ordering trap inside each cluster. A Constraint is a custom resource whose type is created by its ConstraintTemplate, so on a fresh cluster the Constraint's kind does not exist yet and the sync fails with unable to recognize "allowed-registries.yaml": no matches for kind "K8sAllowedRegistries" in version "constraints.gatekeeper.sh/v1beta1". Annotate the templates with argocd.argoproj.io/sync-wave: "0" and the constraints with "1", and add argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true to the constraints so the pre-sync check stops objecting to a type that is about to exist.

# Which pin is each cluster on, and did it land?
argocd app list -l policy-managed=true -o json \
  | jq -r '.[] | [.metadata.name, .spec.destination.name,
                  .spec.source.targetRevision,
                  .status.sync.status, .status.health.status] | @tsv'

policy-canary-1	canary-1	policy-bundle-2026.07.24	Synced	Healthy
policy-nonprod-eu-1	nonprod-eu-1	policy-bundle-2026.07.24	Synced	Healthy
policy-prod-eu-3	prod-eu-3	policy-bundle-2026.07.24	OutOfSync	Progressing
policy-prod-us-1	prod-us-1	policy-bundle-2026.07.17	Synced	Healthy

That is a rollout caught mid-flight. Waves 0 and 1 have landed the new tag, one production cluster is partway through it, and the last one is still on the old pin because its step has not started. A silently stalled wave produces output that looks identical, so have the promotion job compare pins per wave and fail loudly when a cluster is still sitting on an old one past its window.

Prove the Policy Actually Landed

A fire alarm nobody tests is a decoration. Give every cluster a small namespace whose only job is to be a test subject, with two workloads that live in Git next to the policy: one that must always be admitted, and one that must always be rejected. After every promotion, apply both. The known-good workload proves you have not broken ordinary deploys. The known-bad workload proves the rule is switched on, which is the claim nothing else in your pipeline actually checks.

kubectl apply -f canary/known-good.yaml    # image from the internal registry
kubectl apply -f canary/known-bad.yaml     # image from a public registry
echo "exit code: $?"

pod/canary-good created
Error from server (Forbidden): error when creating "canary/known-bad.yaml": admission webhook "validation.gatekeeper.sh" denied the request: [allowed-registries] container <app> has an invalid image repo <docker.io/library/nginx:1.27>, allowed repos are ["registry.internal.example.com/"]
exit code: 1

That is what a working control looks like from the outside. The API server never stored the object; it returned a Status response with code 403 and reason Forbidden, naming the webhook that made the decision, the constraint that fired, and the offending value quoted back. kubectl exits non-zero, so your probe is a five-line shell script in CI rather than a person squinting at a terminal.

Test with a bare Pod on purpose, because a denial reaches you differently when a controller creates the object. Apply a rejected Deployment and the Deployment itself is created happily: the ReplicaSet controller is the one that gets the 403, so nothing appears at your prompt, the Pod count sits at zero, and the message only shows up in kubectl describe replicaset events and in the Deployment's ReplicaFailure condition. Teams who only ever test through a Deployment conclude the policy is off when it is working perfectly.

The failure case is the interesting one. If the known-bad Pod is created, three different problems produce that same result and each needs a different fix. The Constraint may be present but still set to dryrun, because the class overlay that flips it to deny never reached this cluster. The Constraint may be missing entirely, because its ConstraintTemplate failed to compile while the sync agent happily reported success on every other object in the directory. Or the engine may not be consulted at all, because the webhook configuration was deleted during an old incident, or its namespaceSelector excludes the namespace you tested in. One command separates the first two.

kubectl get constraints

NAME                                                                ENFORCEMENT-ACTION   TOTAL-VIOLATIONS
k8sallowedregistries.constraints.gatekeeper.sh/allowed-registries   dryrun               41
k8srequirenonroot.constraints.gatekeeper.sh/require-nonroot         deny                 0

Read those two columns carefully, because they come from different machinery. ENFORCEMENT-ACTION is what admission does to new requests. TOTAL-VIOLATIONS comes from the audit controller, a periodic scan (every sixty seconds by default) that re-checks objects already living in the cluster and writes what it finds into each constraint's status. So 41 is not 41 blocked deployments. It is 41 workloads already running that would fail this rule, which is the number you want before you promote dryrun to deny.

Abort Criteria and the Kill Switch

Decide what "going wrong" looks like before you promote, and write the number down somewhere the promotion job can read it. The most useful signal is the denial rate, because a policy that is about to cause an outage almost always announces itself as a sudden pile of rejected admissions on one cluster while the others stay quiet. Gatekeeper publishes gatekeeper_validation_request_count with an admission_status label, so the query is short. Check your own /metrics output first: recent builds export through OpenTelemetry, which may append _total to the counter name.

curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum by (cluster) (
      rate(gatekeeper_validation_request_count{admission_status="deny"}[5m])
    ) * 60' \
  | jq -r '.data.result[] | "\(.metric.cluster)\t\(.value[1])"'

canary-1	2.4
nonprod-eu-1	0.8
prod-eu-3	61.3

Two clusters are boring and one is not. prod-eu-3 is past the fifty-denials-per-minute line you agreed on, so the wave stops and the pin goes backwards. Rollback is the same move as an application rollback, because the version is a file in Git and nothing else. Assume the commit that pinned production was 3e77b0a.

git revert --no-edit 3e77b0a
git push origin main
argocd app wait policy-prod-eu-3 --sync --timeout 300

[main 4c7a2be] Revert "pin policy bundle 2026.07.24 on prod-eu-3"
 1 file changed, 2 insertions(+), 2 deletions(-)
 
TIMESTAMP                  GROUP        KIND   NAMESPACE   NAME              STATUS   HEALTH
2026-07-24T14:21:07+00:00  argoproj.io  App    argocd      policy-prod-eu-3  Synced   Healthy
 
Name:          argocd/policy-prod-eu-3
Sync Status:   Synced to policy-bundle-2026.07.17 (a91c4e2)
Health Status: Healthy

Because you pinned a digest, "the previous version" is byte-for-byte what ran last week rather than whatever a moving tag happens to point at today, and because OPA swaps bundles in place there is no restart and no gap where the cluster enforces nothing. Practise this before you need it. An untested rollback path is a plan, and plans do badly at 3am.

You also need something faster than a Git round trip for the night a false positive is blocking incident response. Build that switch deliberately, the way a building has marked fire exits rather than a window someone breaks. Gatekeeper honours the label admission.gatekeeper.sh/ignore on a namespace, but only when that namespace also appears in the controller's --exempt-namespace flag, which is exactly the property you want: the escape hatch exists, and the set of places it can be used was decided in advance, in Git, by people who were calm. Kyverno's equivalent is flipping the rule's validate.failureAction from Enforce to Audit. Log every use and open a ticket automatically. The switch you refuse to design gets invented under pressure as kubectl delete validatingwebhookconfiguration, which turns off admission control for the entire cluster and leaves no trace of what it used to contain.

One more decision belongs to the class rather than to the incident: what the API server should do when the policy engine cannot answer at all. That is the webhook's failurePolicy. Ignore admits the request, so deploys keep flowing and unreviewed workloads walk in behind the outage. Fail rejects it, so nothing unreviewed gets in, and an engine outage becomes a deployment outage, including for the deploy that would have fixed the engine. Gatekeeper ships with Ignore and Kyverno ships with Fail, which catches out teams running both engines in the same estate. Choose per class, write the choice into the class overlay, and tell your incident responders which clusters do which, because it is the first thing they will guess wrong while the pager is going off.

Record each promotion in four lines: bundle digest, waves cleared, denial rates observed, and who approved. Keep policy promotions on the same release train as the rest of the platform rather than shipping them whenever a pull request happens to merge. A policy change landing on Friday afternoon carries the same risk as an untested chart landing on Friday afternoon, with fewer people around to notice.

Try This

Sketch the layout in a scratch repository so the shape is in your fingers before it is in production. The promotion file is the artifact worth arguing about, because it is where the abort number and the soak time stop being opinions.

mkdir -p policy-fleet/clusters/{canary-1,nonprod-eu-1,prod-eu-3}
printf '%s\n' 'waves: [canary, nonprod, prod]' \
  'soak_minutes: 120' \
  'abort_if_denies_per_min: 50' \
  'bundle_tag: 2026.07.24' \
  'bundle_digest: sha256:6b19f8b0a2c4d1e7f0a9c3b5d8e2f4a6c9b1d3e5f7a9c2b4d6e8f0a2c4b6d8e0' \
  > policy-fleet/promotion.yaml
cat policy-fleet/promotion.yaml

waves: [canary, nonprod, prod]
soak_minutes: 120
abort_if_denies_per_min: 50
bundle_tag: 2026.07.24
bundle_digest: sha256:6b19f8b0a2c4d1e7f0a9c3b5d8e2f4a6c9b1d3e5f7a9c2b4d6e8f0a2c4b6d8e0

Then answer one question out loud for every class you run: when the policy engine is unreachable, does that cluster admit or reject? If you cannot answer it for the class holding card data, that is the first thing to fix on Monday. The next lesson turns all of this into numbers, including how much of the fleet a rule really covers and whether it has earned the trouble it causes.

Takeaway

The trap worth remembering here: fifty forever-exceptions is fifty policies. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related