Self-healing

What happens when something crashes.

Self-healing means Kubernetes restarts or replaces work without you. A crashed container is restarted. A dead pod copy is replaced. If a machine dies, its work is started on a healthy node. Nobody gets paged at 3am for those cases. A container is the process plus the files it needs; the cluster is the pool of machines that run your apps.

What breaks, and who fixes it

Self-healing runs at three levels, all from the same idea. You write down what you want. The cluster keeps comparing reality to that. The written target is desired state. Say you asked for three copies. Kubernetes keeps counting running copies against three and closes a gap as soon as one opens.

Level one: a container crashes. Every worker (a node) runs a kubelet. The kubelet watches containers on that node and restarts one that dies, in place. RESTARTS ticks up. Level two: a whole Pod dies or is deleted. A Pod is the smallest thing Kubernetes runs. A Deployment records how many copies you want and creates a ReplicaSet whose job is to keep that count. It sees two where three should be, and a replacement is usually up within seconds. Level three: a machine fails. Pods that were on it are recreated on healthy machines, and the app keeps serving from there.

None of that needs a human. That is why a single Pod is disposable. The older pattern kept one server alive for years; losing it meant a restore from backups. Kubernetes keeps the app up by replacing copies, not by making any one copy immortal.

Watch it heal for yourself

Here is a whole Deployment you can apply and poke at. It is written in YAML, a plain-text format that uses indentation instead of brackets to show what belongs to what. Save it as hello.yaml. It asks for three copies of a small web server. The replicas: 3 line is the entire promise, and everything under template describes what one copy looks like.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80

Send it to the cluster with kubectl, the command-line tool you use to talk to Kubernetes, then list the Pods it made.

kubectl apply -f hello.yaml
kubectl get pods -l app=hello

deployment.apps/hello created
 
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c8f5b4-2xk9p    1/1     Running   0          18s
hello-7d9c8f5b4-8vq4m    1/1     Running   0          18s
hello-7d9c8f5b4-lr7cd    1/1     Running   0          18s

Three copies, all Running. Now break one on purpose. Pick any Pod name from that list, delete it, and watch what the cluster does next. The -w flag on the second command means watch, so it keeps printing changes live instead of answering once and quitting.

kubectl delete pod hello-7d9c8f5b4-2xk9p
kubectl get pods -l app=hello -w

pod "hello-7d9c8f5b4-2xk9p" deleted
 
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c8f5b4-8vq4m    1/1     Running   0          95s
hello-7d9c8f5b4-lr7cd    1/1     Running   0          95s
hello-7d9c8f5b4-qm2ft    1/1     Running   0          3s     <-- brand-new, count restored to 3

You deleted a Pod. A new one showed up under a different name and the count went back to three. You never ran a create command for it. The ReplicaSet saw two where three should be and closed the gap by itself. That gap-closing is the whole trick, and you watched it happen live. The same loop runs for every kind of failure, whether you broke something by hand or a real crash did it for you.

When healing runs out of road

Deleting a Pod is a polite kind of failure, because the replacement comes up healthy. The failure you hit on your first real app is messier: a container that dies the instant it starts, usually from a wrong image tag, a missing file, or a setting pointing at nothing. Here is a Deployment rigged to fail exactly that way. Its container prints one line, waits two seconds, then exits with an error. Then it does the same thing again.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crasher
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crasher
  template:
    metadata:
      labels:
        app: crasher
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "echo starting; sleep 2; exit 1"]

Apply it, give it a minute to fail a few times, and list it.

kubectl apply -f crasher.yaml
kubectl get pods -l app=crasher

deployment.apps/crasher created
 
NAME                       READY   STATUS             RESTARTS      AGE
crasher-6b9f7c4d8-w2m4k    0/1     CrashLoopBackOff   5 (46s ago)   3m1s

READY says 0/1 and the STATUS is CrashLoopBackOff, the single most common thing a beginner watches go wrong. The kubelet started the container, it died, the kubelet started it again, and RESTARTS has already reached 5. To find out why, ask the Pod to describe itself.

kubectl describe pod crasher-6b9f7c4d8-w2m4k

Containers:
  app:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Restart Count:  5
...
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Normal   Pulled   3m (x5 over 3m)       kubelet  Container image "busybox:1.36" already present on machine
  Warning  BackOff  15s (x12 over 2m45s)  kubelet  Back-off restarting failed container app

The Last State line is the tell. The container Terminated with Exit Code 1, which means the program inside failed on its own. Down in Events you can watch the kubelet at work, Back-off restarting failed container, over and over. Backoff is the catch. Each retry waits longer than the one before it, ten seconds, then twenty, then up to five minutes, so a broken Pod does not hammer the node forever. And here is the honest limit of self-healing. Restarting cannot fix a bug in your code, a file that is not there, or an image tag you typed wrong. Kubernetes will loop on it patiently, but a Pod stuck in CrashLoopBackOff is the cluster telling you it tried and cannot. Run kubectl logs crasher-6b9f7c4d8-w2m4k --previous to read the dead container's last words, fix the real cause, then apply again. (Point a Deployment at an image name that does not exist and you meet the close cousin of this state, ImagePullBackOff.)

Teaching the cluster what healthy means

So far Kubernetes has healed things that died or never started. There is a blind spot. By default it only knows whether the container process is running, not whether the app is useful. A web server can be frozen, or answering every request with an error, and the Pod still shows Running, so nothing is restarted. A probe is a health check the cluster repeats. A liveness probe asks whether the copy is still alive; if it fails, Kubernetes restarts that container even though the process never exited. A readiness probe asks whether it should receive traffic; if not, the cluster stops sending users there until it passes. Both are in the next lesson. A Deployment already restarts crashes and replaces pods on a dead node without those probes.

Self-healing is not magic, and it pays to know which piece does what. Controllers recreate Pods that vanish. Kubelets restart containers that exit. Neither of them reads your code. A process that crashes in a loop turns healing into CrashLoopBackOff, which is noise rather than health. When a restart counter climbs, read it as an alarm and go fix the crash or the failing liveness probe behind it.

Losing a whole node is the slowest heal of the three. The cluster waits out a built-in grace timer (Kubernetes calls it a toleration timeout) before it accepts the machine is really gone, and only then do its Pods get rescheduled elsewhere. Stateless Deployments come back so smoothly that healing looks free. Anything holding data asks more of you: persistent volumes so the data outlives the Pod, and a stable identity so the replacement knows which copy it is meant to be.

During a real incident, deleting a sick Pod is a fair nudge, as long as something owns it and will make another one. Delete a bare Pod, one created on its own with no Deployment or ReplicaSet behind it, and nothing brings it back. You have taken your own service down by hand. Check the ownerReferences field on a Pod first and you will know which of the two you are holding.

Try this

Start a Deployment, delete one of its Pods, and watch the replacement arrive. Then kill the process inside a container and watch the restart counter move.

$ kubectl create deployment heal --image=nginx:1.27 --replicas=2
deployment.apps/heal created
 
$ POD=$(kubectl get pod -l app=heal -o jsonpath='{.items[0].metadata.name}')
$ kubectl delete pod $POD
pod "heal-…" deleted
 
$ kubectl get pods -l app=heal -w
NAME       READY   STATUS              RESTARTS   AGE
heal-…     1/1     Running             0          30s
heal-…     0/1     ContainerCreating   0          1s
heal-…     1/1     Running             0          3s
# Ctrl+C
 
$ kubectl exec deploy/heal -- nginx -s stop
$ kubectl get pods -l app=heal
NAME     READY   STATUS    RESTARTS   AGE
heal-…   1/1     Running   1          55s
heal-…   1/1     Running   0          40s
 
$ kubectl delete deployment heal
deployment.apps "heal" deleted

Takeaway

Controllers replace missing Pods. Kubelets restart exited containers. That covers a crash or a dead machine the moment it happens, with nothing from you. A RESTARTS count is only good news the first time it moves. Keep watching it climb and the cluster has done everything it can, which leaves the fix to you: read the logs, check the image tag, or write the liveness probe that turns a frozen app into one Kubernetes can actually see.

Related