>_SecOpsLog_
CoursesSecrets management foundationsKubernetes secrets, honestly

Kubernetes secrets, honestly

Base64 is not encryption; etcd is the real story.

Beginner25 min · lesson 7 of 13

Kubernetes, the system that decides which containers run on which machines, ships a built-in object type called a Secret. The name promises more than the thing delivers. A Secret is a sealed envelope, not a safe. It keeps the password out of casual view, and anyone allowed to hold the envelope can open it with one command. Plenty of real incidents start with a team reading the word "Secret," hearing "encrypted," and switching their brain off.

So what is it, exactly? A small API object, meaning a record the cluster stores and hands out: a named bag of key/value pairs holding something like a database password, an API (application programming interface, how one program talks to another) token, or a TLS (transport layer security, the padlock behind https) private key. The cluster keeps one copy centrally and delivers it to the pods that ask for it. A pod is the smallest thing Kubernetes runs, one or more containers sharing a network address, and it gets the value either as a file mounted inside the container or as an environment variable. That is genuinely useful. It kills the crudest failure of all, credentials baked into a container image where anyone who can pull that image can read them, and it puts access behind the cluster's permission system. What a Secret does not do, on a default cluster, is encrypt anything.

Base64 is encoding, not encryption

Secret values sit in the object base64-encoded. Base64 is a reversible way of writing bytes down using only letters, digits, + and /, so that any blob survives being pasted into a YAML (a plain-text config format) or JSON file. There is no key. There is no algorithm to pick. Reversing it takes one command, base64 -d, and no permission of any kind:

shell
$ kubectl create secret generic db-creds --from-literal=password='S3cr3t!'
secret/db-creds created
 
$ kubectl get secret db-creds -o jsonpath='{.data.password}'
UzNjcjN0IQ==
 
$ kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
S3cr3t!

Base64 is here for a boring plumbing reason. A Secret value might be binary, a TLS key or a Java keystore, and binary bytes do not paste cleanly into YAML. That is the whole story. Read "base64-encoded" as *plaintext with one extra keystroke*. Anything that can read the object can read the value.

Where the bytes actually live

The life of a Secret
1kubectl apply
Secret manifest
2kube-apiserver
validates, persists
3etcd
plaintext by default
4kubelet
tmpfs on the node
5container
file or env var
Every hop is somewhere the value can be read: the API server (RBAC), etcd (disk and backups), the node (root), the container (exec, crash dumps).

Follow the value. kubectl hands the Secret to the kube-apiserver (the cluster's front door, the one process everything else talks to), which writes it into etcd, the key-value database holding all cluster state. On a default cluster it lands there unencrypted. It is not even base64-wrapped: the API decodes the base64 and writes the raw bytes into etcd's binary serialization format, recoverable exactly, as the hexdump below shows. Later, when a pod that references the Secret gets scheduled onto a machine, the kubelet (the Kubernetes agent running on every node) pulls the value down and materializes it for the container, either as a file on a *tmpfs* (a filesystem that lives in RAM and never touches the node's disk) or injected into the process environment. Do not take the etcd claim on faith. Check it yourself on a control-plane node:

shell
# on a kubeadm control-plane node
$ sudo ETCDCTL_API=3 etcdctl \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert   /etc/kubernetes/pki/etcd/server.crt \
    --key    /etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/default/db-creds | hexdump -C | grep -A1 password
00000090  70 61 73 73 77 6f 72 64  12 07 53 33 63 72 33 74  |password..S3cr3t|
000000a0  21 1a 06 4f 70 61 71 75  65                       |!..Opaque|
# the password is sitting in the database — no decryption happened

The blast radius is wider than it looks. etcd's data directory sits on the control-plane disk. etcd snapshots and cluster backups contain every secret in the cluster, not one namespace's worth. So whoever can read a control-plane filesystem, or the storage bucket where last night's backup landed, holds all of them at once. At scale that is usually the real exposure. One over-shared backup bucket beats any number of carefully permissioned namespaces.

Turning on encryption at rest

Kubernetes can encrypt Secrets on the way into etcd. You give the kube-apiserver an EncryptionConfiguration file that lists providers in order. aescbc and aesgcm encrypt with a key stored locally on the control plane. kms (key management service, an outside box that holds keys for you) hands the work to something like AWS KMS or Vault, and you want API version v2, stable since Kubernetes 1.29. identity means "write it in plaintext." Order matters: the first provider encrypts every new write, and the ones after it exist so data written under an older scheme still reads back.

/etc/kubernetes/enc/enc.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: 5FyIMHmYIm5XJ2fWCqA0M9TgUpwsPS62nBv+CWJ7RVA=  # head -c 32 /dev/urandom | base64
      - identity: {}   # fallback: read pre-existing plaintext data
shell
# add to the kube-apiserver static pod manifest, then let it restart:
#   --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
 
# rewrite every existing secret so it gets encrypted on the way back in
$ kubectl get secrets -A -o json | kubectl replace -f -
secret/db-creds replaced
secret/registry-pull replaced
...
 
# verify: the stored value now carries an encryption header
$ sudo ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
    --cacert ... --cert ... --key ... | head -2
/registry/secrets/default/db-creds
k8s:enc:aescbc:v1:key1:...binary ciphertext...

Two things stop this from being the whole answer. With aescbc, the encryption key is a file sitting on the same control-plane node as etcd. You have locked the safe and taped the key to its door, which mostly helps against *offline* copies: stolen disks, snapshots, backups in a bucket. A kms provider fixes that by keeping the key in a service that can refuse to hand it over. On managed clusters you often get that already. EKS, GKE and AKS (the Amazon, Google and Microsoft hosted Kubernetes services) encrypt etcd with cloud KMS keys either by default or after one setting. Go and check which one applies to your cluster rather than assuming.

Who can read a Secret

The other boundary is RBAC (role-based access control), the rulebook saying which users and service accounts may perform which verbs, like get or create, on which kinds of object. Secrets come with a catch. The verbs get, list and watch all return the full value, so a role you would casually call "read-only on secrets" is a role that discloses plaintext. Find out who holds those three verbs. Where the list is short, pin roles to named secrets with resourceNames instead of the whole resource:

shell
$ kubectl auth can-i get secrets -n prod \
    --as=system:serviceaccount:prod:ci-runner
no
 
$ kubectl auth can-i list secrets -n prod --as=jenny@corp.example
yes   # jenny can read every secret value in prod
If you can create pods, you can read secrets
RBAC on the secrets resource is only half the fence. Anyone, a person or a controller, with create on pods in a namespace can launch a pod that mounts any secret in that namespace and prints it. No get secrets needed. exec into a pod that already mounts one gets you the same thing. When you audit who can see a credential, count pod-create and pod-exec rights as secret-read rights, and treat a namespace as one trust zone. Parking a low-sensitivity app next to a high-sensitivity one in the same namespace quietly merges their secrets.

How the container receives the value matters too. Mount Secrets as files rather than environment variables. Env vars leak through crash dumps, get inherited by every child process, and turn up in debug endpoints (this course's first lesson walked through the mechanics), and changing one means restarting the pod, while a mounted Secret file updates in place within about a minute. Set defaultMode: 0400 on the mount so only the owning user can read it, and mark rarely-changing values immutable: true. That blocks silent edits and cuts apiserver watch load in large clusters.

What Secrets don't do, and where this goes

Even fully hardened, the built-in object is a place to put a value, not a system for managing one. There is no rotation. Nothing expires, nothing regenerates, nothing tells the app a value changed. There is no version history and no usage trail: API audit logs record who touched the object, never which process used which credential. Secrets are scoped to a single namespace, so one credential shared across fifty namespaces is fifty copies that will drift apart. Those gaps (rotation, auditability, short-lived credentials minted on demand) are the argument for an external secret manager, which the final lesson of this course weighs up.

There is a nearer problem to handle first. A Secret manifest is YAML, and in a GitOps workflow (where the desired state of the cluster lives in a git repository and a controller applies whatever is there) that YAML lives in git. The previous lesson showed what git does with anything you commit: it keeps it, in history, in every clone, effectively forever. Base64 in git is plaintext in git. The next lesson covers the two standard escapes. SOPS encrypts the values inside the file you commit. sealed-secrets encrypts a Secret so that exactly one cluster can ever open it.

Here is how this goes wrong in practice. Someone copies etcd snapshots into a shared S3 bucket (Amazon's file storage service) "temporarily" for a migration. Six months later the migration is finished and the bucket policy is still too open. Encryption at rest with a cloud KMS key would have turned that dump into boring ciphertext. A local aescbc key living on the same disk as etcd gives you far less, because anyone who can read that disk can read the key sitting next to it. Pick the provider that matches the threat you actually face: someone walking off with a disk, or someone abusing a live API.

RBAC reviews go wrong when they only look at ClusterRoles with "admin" in the name. A controller that can create pods in a namespace inherits every Secret in it. A CI (continuous integration, the system that builds and ships your code) service account that once needed to deploy to prod usually still can, years later. Pin resourceNames where you can, split namespaces along trust lines, and treat exec into a pod that mounts a Secret as reading the Secret.

Default to file mounts with defaultMode 0400, plus immutable: true on values that rarely change. Env injection is convenient for twelve-factor apps and awful in crash dumps and child processes. When a value has to change, pair the Secret update with a rollout so the pods pick it up. Better still, move that credential into a manager that can mint a short-lived replacement on demand.

In a shared cluster, treat every namespace as a trust boundary you drew on purpose. Apps that should not share Secrets should not share a namespace, and your CI deployer should not land beside customer workloads because a Helm chart made that the easy path. Encryption at rest without RBAC hygiene is a locked filing cabinet in an office nobody locks.

Try this

On a kind or minikube cluster of your own (both run a throwaway Kubernetes cluster on your laptop), create a Secret, decode it, then ask RBAC whether a service account can list secrets. You are getting a feel for the envelope here. Nobody is encrypting production etcd yet.

terminal
kubectl create secret generic db-creds --from-literal=password='S3cr3t!' --dry-run=client -o yaml | kubectl apply -f -
kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d; echo
kubectl auth can-i list secrets -n default --as=system:serviceaccount:default:default
output
secret/db-creds created
S3cr3t!
yes
# base64 peeled with one pipe; default SA often still too wide — tighten before prod

Takeaway

A Kubernetes Secret is a delivery mechanism with a permission check attached, not a vault. Base64 is encoding. etcd and the backups of etcd are where the value really lives. Pod-create rights are secret-read rights.

Next time you touch a cluster: turn on encryption at rest and prefer a KMS provider, list who holds get and list on secrets plus who can create pods, and move your env-var Secrets to file mounts with tight modes.

Quick check
01A service account in the prod namespace has lost get, list and watch on secrets. It can still create pods. Can it read the secret values in prod?
Correct — RBAC on the secrets resource is only half the fence: create on pods lets you mount and dump any secret in that namespace without ever holding get secrets.
Incorrect — Not so. This is the exact misreading the lesson pushes back on. Verbs on the secrets resource are one boundary, and pod-create walks around them.
Incorrect — Close, but exec is an extra route rather than the only one. Create on pods is enough by itself, because you can start a fresh pod that mounts the secret.
Incorrect — No. Encryption at rest protects what gets written to etcd. The kubelet still decrypts the value and hands it to any pod that mounts it, so the mounting pod sees plaintext.
02You switch on aescbc encryption at rest. What does that actually protect, and what does it leave alone?
Correct — Offline copies of etcd get much harder to read, while a stolen key file or a live pod with the Secret mounted still gives the value up.
Incorrect — No. Encryption at rest guards storage. It makes no authorization decision about mounts.
Incorrect — No. The API still returns Secret data to any client allowed to ask. What changed is how the bytes sit in etcd.
Incorrect — No. Built-in Secrets rotate nothing. Sealed-secrets key renewal is a separate mechanism that does something else.
03You pointed the kube-apiserver at an EncryptionConfiguration with aescbc first and identity as the fallback, and the apiserver restarted cleanly. You read db-creds straight out of etcd to check, and instead of a k8s:enc:aescbc:v1:key1: header you still see the password in the clear. What do you do next?
Correct — The first provider in the list encrypts every new write, and nothing else. Objects written before the restart keep sitting there exactly as they were until something writes them again, which is why the lesson pipes kubectl get secrets -A -o json into kubectl replace -f - and only then re-checks etcd.
Incorrect — No. identity means write it in plaintext, and as a fallback it exists so data written under an older scheme still reads back. Removing it does not rewrite anything, and it takes away the only provider that can still read your not-yet-rewritten Secrets.
Incorrect — No, on both counts. Base64 is encoding, not encryption. And the API decodes the base64 before writing, so what you are staring at is the raw password in etcd's serialization format, which is the pre-encryption state.
Incorrect — No. Every provider only encrypts on write, so the old rows stay plaintext under kms too until you rewrite them. A kms provider is worth having for a different reason: it keeps the key in a service that can refuse to hand it over, rather than in a file next to etcd.
Images, build args, and layersDetecting secrets before merge

Related

Why production Vault existsVault from dev to productionThe secrets threat model & secret-zeroAdvanced secrets management