cgroups v2 and resource containment

Memory, CPU and PID limits, OOM kills and where to read them.

A batch worker has --memory 64m. After a run, docker inspect reports OOMKilled=true and ExitCode=0. The engineer on call files it as a Docker bug, since a container that was killed for memory can't have exited cleanly. Both values are correct. To read them correctly, you need the mechanism underneath: control groups, or cgroups, the kernel feature that counts and limits what a group of processes uses. Namespaces, from the previous lesson, decide what a container can see. cgroups decide how much memory, CPU time and how many processes it can take.

You'll run everything on the main lab VM, and it stays inside lab-* containers. One small image, lab-stress:1, provides a load generator: its Dockerfile is in the lesson files (copy it to ~/lab/cgroups).

# lab-stress:1, a small load generator for the cgroups lesson (stress-ng from Alpine's community repo)
FROM alpine:3.24
RUN apk add --no-cache stress-ng
ENTRYPOINT ["stress-ng"]

stress-ng is maintained and packaged in Alpine. The progrium/stress image that older tutorials use is stored in the Docker schema 1 manifest format. Releases of containerd from 2.1 on refuse to pull that format, so the image no longer works on Docker 29.

One tree, one scope per container

Build the image in ~/lab/cgroups:

Then ask Docker and the kernel which cgroup setup this host has:

This track uses cgroup v2 only: one unified hierarchy mounted at /sys/fs/cgroup (filesystem type cgroup2fs). Current Ubuntu, Debian, Fedora and RHEL releases use it by default. driver systemd means Docker asks systemd to create each container's cgroup as a transient unit called docker-<container id>.scope under system.slice. So systemd's tools see containers like any other unit. The VM has no swap (Swap: 0B), which matters for the swap section below.

Limits are files the kernel reads

Start a container with a memory limit, a CPU limit and a process limit, then read its cgroup directory on the host:

memory.current is what the group uses now (about 5 MiB for an idle nginx; yours will differ) and pids.current counts its tasks, the master and four workers. The swap line needs care. cgroup v2 has separate files for memory and swap. Docker's API still takes one combined number, MemorySwap, which defaults to twice --memory when you don't set it (536870912 above). The runtime writes the difference into memory.swap.max. So a plain --memory 256m allows up to another 256 MiB of swap on a host that has swap. The same values also appear on the systemd unit, because systemd created the scope:

/proc/<pid>/cgroup holds a single 0:: line on cgroup v2: the process's path in the unified tree. systemd-cgls -u prints the processes in one unit. The full tree (systemd-cgls without arguments) also shows containerd and the shims inside containerd.service and dockerd inside docker.service, separate from the container scopes.

What the container sees, and what it does not

From inside the container, you read the container's own limit from /sys/fs/cgroup/memory.max, because its cgroup namespace shows its own cgroup as the root. /proc/meminfo and nproc aren't cgroup-aware: they report the VM's 6 GB and 4 CPUs. A program that sizes a heap, a cache or a thread pool from those numbers plans for resources it'll never get. It's then killed or throttled when it grows into them. Current Java (container support is on by default since JDK 10) and Go (GOMAXPROCS follows the CPU limit since Go 1.25) read the cgroup files. Check what your runtime and your own scripts read.

Setting --memory-swap equal to --memory gives the container no swap at all:

On a host with swap, a container that reaches memory.max without that setting starts swapping instead of being killed. memory.current never goes above memory.max; the swapped pages appear in memory.swap.current. So docker stats keeps showing usage at or under the limit while the container slows down. The lab VM has no swap, so the swap allowance here is a number in a file with no effect, and you won't see swapping in this lab.

A container with no limits has no memory or CPU cap, but it does have a process cap:

max means no limit and cpu.max reads max 100000 (no quota, 100 ms period). pids.max is 6326 because the systemd driver gives every scope systemd's DefaultTasksMax, 15 percent of the kernel's task limit on this VM. A runaway process tree in an "unlimited" container therefore stops at a few thousand tasks instead of exhausting the host. But a few thousand per container is still enough to hurt a host running many of them, so set --pids-limit explicitly. Choosing values for real services is part of "Production best practices" in Docker in depth.

Out of memory

Give a container 64 MiB and make it hold 200 MiB. head writes 200 MiB of zeros into a pipe, and tail must keep all of it in memory because the input has no line break:

Once the kernel can't reclaim enough memory to keep the group under memory.max, the OOM killer picks a process inside that group. It sends it SIGKILL, which can't be caught. The likely victim is tail, the process holding the memory. The container exited 137 (128 plus signal 9) and printed nothing. An earlier run of the same lab also showed busybox sh printing Killed. The daemon saw the kernel's OOM notification and set State.OOMKilled. docker events replays the oom event followed by die with exit code 137. In production you'd watch the same stream live (docker events --filter event=oom) or alert on the counter below.

The cgroup itself keeps the count in memory.events, which only exists while the container does. This time stress-ng asks for 200 MiB inside a 64 MiB limit and is left running:

In memory.events, max counts how often usage hit the limit, oom how often the OOM killer was invoked, and oom_kill how many processes it killed; your counts will differ. The kills landed on stress-ng's worker processes. stress-ng started a new worker after each one and finished its 15-second run normally. The container exited 0, and OOMKilled is still true, because it means "an OOM kill happened in this container's cgroup", not "the main process died of it". That's the worker from the opening. Read OOMKilled together with the exit code. For a long-running container, alert on oom_kill increasing: a web server that loses a worker process to the OOM killer can keep running and never set a non-zero exit code.

CPU: throttled, not killed

A CPU limit never kills anything. When the group's used its quota for the current period, its threads wait until the next one. Two busy stress-ng workers under --cpus 0.5:

cpu.max is 50000 100000: 50 ms per 100 ms period. In all 51 of the first 51 periods the group used its quota and was throttled. throttled_usec adds up the waiting: about 8 seconds across both workers in 5 seconds of wall time. If you see a latency problem with clean logs and a climbing nr_throttled, that's a CPU limit doing its job. systemd-cgtop gives the per-cgroup view the way top gives the per-process one; in batch mode, the columns are path, tasks, CPU percent, memory, and input and output per second:

The first sample has no CPU figure, because cgtop needs two readings to compute a rate. Both tools take a short sample, and here they read 49 and 55 percent for lab-cpu, either side of its 50 percent cap. A sample that lines up badly with the 100 ms periods reads high or low. For a precise figure, use cpu.stat: usage_usec above is 2.6 seconds of CPU in about 5 seconds, close to the cap. docker stats also prints each container's memory limit: 256 MiB for lab-web, and the VM's whole memory for lab-cpu, which has none.

A fork bomb against pids.max

b(){ b|b& };b is a fork bomb written for busybox sh: each call starts two more in the background, forever. With --pids-limit 100 it fills its own group and nothing else:

pids.current sits at the limit and the max line in pids.events counts the forks the kernel refused (54 in five seconds here). Other containers and the host keep their process slots. The --cpus 0.5 on the bomb matters too: the failing fork loop still burns CPU, and without a CPU limit it would take every core it could get. docker rm -f stops it in one step: it kills PID 1, and when the first process of a PID namespace dies the kernel kills every other process in it:

Limits like these contain a noisy or broken workload. They aren't a security boundary. A container that stays under every limit can still attack the kernel through system calls and capabilities, which "Capabilities, cap-drop and no-new-privileges" takes up next.

Try this

On the main lab VM, rerun the lab-cpu stress step with --cpus 1.5 instead of --cpus 0.5, keeping the two workers. Predict the cpu.max line and whether nr_throttled still climbs, then compare cpu.stat with the 51-of-51 reading above. After that, rerun the head -c 200m /dev/zero | tail pipeline with --memory 256m in place of 64m, and check OOMKilled and ExitCode with the same docker inspect format. Remove both containers when you're done.

Bottom line

Give every container explicit --memory, --cpus and --pids-limit values, and read OOMKilled together with the exit code. The stress-ng run showed OOMKilled=true with exit code 0, because the kills hit workers, not PID 1, and an alert on non-zero exits alone misses that. A container started with no limits isn't unbounded on pids either: it inherits the systemd default of 6326 tasks, a number nobody on your team chose.

Related