Kubernetes incident response

Cut identity, preserve, hunt persistence, trust.

When detection fires on a real cluster compromise, the response is the familiar race: contain, understand, and evict without destroying the evidence. Kubernetes adds its own wrinkles — identities to revoke, persistence to hunt in RBAC and admission, and the question of whether trust itself was broken.

Contain the identity, preserve the workload

Start by cutting the compromised identity: delete or rotate the affected service account and its tokens so a stolen token stops working, and remove any RBAC the attacker granted. Isolate the affected pod with a deny-all NetworkPolicy and cordon its node so nothing new schedules there, then preserve state — capture the pod logs, the runtime event trail, and a node snapshot — before evicting. As with cloud IR, deleting the pod first destroys the process memory and state that tell you what happened; isolate and snapshot, then remove.

# 1. Cut the identity: rotate/delete the SA and revoke what it can do.
kubectl delete secret compromised-sa-token -n prod       # legacy token
kubectl delete clusterrolebinding attacker-added         # remove their persistence
# 2. Isolate the pod (deny-all NetworkPolicy) and cordon the node.
kubectl label pod evil-7c9 quarantine=true -n prod
kubectl cordon node-17
# 3. Preserve BEFORE evicting.
kubectl logs evil-7c9 -n prod --previous > ir-pod.log     # + node disk snapshot
kubectl delete pod evil-7c9 -n prod --grace-period=0

Hunt persistence, decide on trust, recover

Reconstruct the intrusion from the API audit log and runtime events, then hunt for the persistence Kubernetes attackers favor: new or modified ClusterRoleBindings, mutating/validating webhooks, DaemonSets and CronJobs, tampered images, and altered admission policy. The hardest judgment is trust: if the attacker reached the control plane, etcd, or the cluster CA, forged identities may persist and the only safe recovery is rotating the CA and credentials and rebuilding. Rotate every secret the compromised identity could reach, remove the footholds, rebuild workloads from known-good images, and only then restore service. Rehearse this as a drill so a real cluster incident is a practiced runbook, not an improvisation.