Secrets & encryption at rest

Encrypt etcd; pull from external stores.

A Kubernetes Secret feels protected but is only base64-encoded — encoding, not encryption. Anyone who can read the object, read etcd directly, or get hold of an etcd backup can recover the plaintext instantly. So the Secret object is a distribution mechanism, not a vault, and two things actually protect it: encrypting it at rest in etcd, and tightly restricting who can read it via RBAC.

# "encryption"? no — one command from plaintext
$ kubectl get secret db-creds -n payments -o jsonpath='{.data.password}' \
    | base64 -d
S3cr3tP@ss           # anyone with get on the secret sees this

Encrypt at rest

Encryption at rest makes the API server encrypt secret data before it writes to etcd, driven by an EncryptionConfiguration you pass with --encryption-provider-config. Provider order matters: the first entry is used to encrypt, and any listed provider can decrypt, so you keep identity last as a read fallback while migrating. A KMS v2 provider (key held in Vault or a cloud KMS, outside the cluster) is strongest; aescbc/aesgcm keep the key in the config file, which then becomes as sensitive as the data itself.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
  providers:
  - aescbc:                 # encrypts new writes (first = write provider)
      keys:
      - name: key1
        secret: <32-byte base64 key>
  - identity: {}            # read fallback for still-plaintext secrets
# wire it in: kube-apiserver --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Encrypt what already exists, then verify

Turning encryption on only affects future writes, so existing secrets stay in whatever form they were last written. Force a rewrite of every secret to re-encrypt them, then confirm at the storage layer by reading the raw etcd value — it should begin with k8s:enc:, not a legible base64 string. That etcdctl check is the proof; the flag alone is not.

$ kubectl get secrets -A -o json | kubectl replace -f -   # rewrite = re-encrypt
 
$ sudo ETCDCTL_API=3 etcdctl \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/payments/db-creds | hexdump -C | head
... k8s:enc:aescbc:v1:key1: ...      # not plaintext base64 = encrypted

Consume secrets safely

How a pod reads a secret matters. Prefer a mounted volume (readOnly) over environment variables: env vars leak through /proc, crash dumps, child processes, and the occasional debug log, whereas a mounted file stays put and can be made read-only. For rotation and to keep the source of truth out of etcd entirely, pull from an external store — Vault via the External Secrets Operator or the Secrets Store CSI Driver.

spec:
  containers:
  - name: app
    image: registry.internal/app:1.4.2
    volumeMounts:
    - { name: creds, mountPath: /etc/creds, readOnly: true }   # file, not env
  volumes:
  - name: creds
    secret: { secretName: db-creds }