Runtime incident response
Isolate, preserve, hunt persistence, recover.
When runtime detection fires on a real compromise, the response is a race to contain, understand, and evict the attacker — on infrastructure where a pod can be gone in seconds. As with cloud IR, the winning move is a rehearsed runbook that contains the threat without destroying the evidence you need.
Contain and preserve, in order
The reflex to delete a compromised pod is usually wrong: it discards the process state and memory that tell you what happened. Instead, isolate first — apply a deny-all NetworkPolicy so the pod can neither exfiltrate nor receive commands, and cordon the node so nothing new schedules onto a potentially-compromised host. Then preserve: capture the pod’s logs, the runtime process tree, and a disk/memory snapshot of the node before evicting. Only after evidence is captured do you evict the workload and, if the node itself may be compromised, drain and reimage it.
# 1. Isolate the pod (deny-all NetworkPolicy selecting a quarantine label).kubectl label pod payments-api-7c9 quarantine=true -n prod# 2. Cordon the node so nothing new schedules there.kubectl cordon node-17# 3. Preserve BEFORE evicting: logs, process tree, node snapshot.kubectl logs payments-api-7c9 -n prod --previous > ir-8891-pod.log# (capture Tetragon/Falco events for the pod; snapshot the node disk via the cloud API)# 4. Only now evict — and drain+reimage the node if it may be compromised.kubectl delete pod payments-api-7c9 -n prod --grace-period=0
Investigate, hunt persistence, recover
Reconstruct the intrusion from the runtime event trail (Falco/Tetragon), the Kubernetes audit log, and the node’s captured state — a centralized, tamper-resistant log store means the evidence survives even if the node was owned. Then hunt for persistence, because a competent attacker plants more than one foothold: new or modified DaemonSets and workloads, added RBAC bindings, mounted host paths, tampered images or admission webhooks, and cron or systemd units on the node itself. Rotate any credentials the workload could reach, remove the footholds, rebuild from known-good images, and only then restore service — and rehearse the whole sequence as a drill so the first real run is not improvised.