Configuring the daemon safely

daemon.json, what to leave alone, and how to change it without surprises.

Here's what an ordinary systemctl restart docker does to a running container on a default install:

There's no daemon.json. A fresh Docker 29 install runs entirely on built-in defaults, which the docker info line sums up: containerd image store, json-file logging, live-restore off, data under /var/lib/docker. The restart then stopped lab-web. Exited (137) means nginx ended on SIGKILL (128 + 9), not a clean exit. A run on Engine 29.8.2 showed Exited (0). With live-restore off, dockerd stops every container when it shuts down. So a daemon restart is a restart of every workload on the host. Most daemon changes need a restart, and you're here to make those changes without that kind of surprise. You'll see what belongs in daemon.json, how to change it safely, and which settings behave differently on Docker 29 than older guides say. Don't assume the file is missing on your own VMs. On a link with an MTU below 1500 (typical behind a VPN), the lab ka daemon.json with mtu settings when it creates your VM, and the procedure below keeps them.

What goes in daemon.json

/etc/docker/daemon.json sets host-wide defaults for dockerd. Container flags still override many of them, but every container that doesn't say otherwise inherits them. A small file is a good file. This one sets the three things most production hosts want:

{
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "default-address-pools": [
    { "base": "10.200.0.0/16", "size": 24 }
  ]
}

You'll meet other keys too. registry-mirrors sets a pull-through cache. features holds feature switches; on Docker 29 the one that matters is containerd-snapshotter, shown below. firewall-backend picks iptables or nftables, which "Publishing ports and the packet path" covers. userns-remap turns on user-namespace remapping and, on Docker 29, disables the containerd image store; "User-namespace remapping" in Advanced container security explains that trade. Security defaults live here too. no-new-privileges applies to every new container. seccomp-profile is a path to the seccomp profile used instead of the built-in one ("seccomp: filter system calls" in Advanced container security shows how to build one from the default).

Validate before you touch the live file

dockerd --validate --config-file parses a file and checks every key against the daemon's options without starting anything. Run it on the new file where you edited it, before it replaces the live one:

{
  "live-restore": true,
  "log-driver": "json-file",
}

{
  "live-restore": true,
  "log-opt": { "max-size": "10m" }
}

The trailing comma is a JSON syntax error, reported with the character the parser choked on. The second file is valid JSON with a misspelled key: log-opt instead of log-opts. The validator flattens unknown objects, so the message names the nested key max-size rather than the misspelled parent. When a key you know is valid is reported as unknown, check the spelling of the object around it. Both exit with status 1. That makes dockerd --validate easy to use as a gate in Ansible, Puppet or a CI check of the file.

Back up, merge, validate, install, reload, restart, verify

Never write a new daemon.json over the old one blindly. Something may already be in it: the kit's MTU settings, a registry mirror, a log default someone relies on. And a rollback needs the exact previous file. Here's the procedure:

Other lessons in this course that touch daemon.json on the sec VM use the same steps. Some write the merged file straight to /etc/docker/daemon.json and validate it there before the restart.

The backup holds {} because this VM had no file. jq -s reads both files into an array, and * merges the two objects recursively. So nested objects such as log-opts combine instead of replacing each other, and on a key present in both, the lesson's file wins. The merged result is validated where it was written, before it replaces the live file.

systemctl reload docker sends SIGHUP to dockerd. The daemon re-reads daemon.json and applies the subset of options documented as reloadable, without stopping anything. live-restore is one of them, and docker info now reports live-restore=true. Log options aren't on the reloadable list, so they wait for the restart (the lab-new container below shows them arriving). That order matters on a host that's never had live-restore. Reload first so live-restore is active, then restart for the remaining keys.

MainPID changed, so dockerd really restarted. The PID of lab-web's shim stayed the same, which proves the container process was never stopped. The short Up time in docker ps counts from the docker start a few seconds earlier, so it says little either way. "Engine architecture on Docker 29" showed that the shim is the container's parent. It kept the process alive while the daemon was gone, and the new dockerd reconnected to it. Live-restore covers daemon restarts and patch-level upgrades. It doesn't survive a host reboot, and it doesn't apply to Swarm services. The docs also warn that it may fail when settings such as the bridge IP or the storage driver change between restarts. A long outage can fill the container's output pipe, too, and block the application's writes until dockerd returns.

Defaults reach new containers only

lab-new, created after the restart, carries max-file: 3 and max-size: 10m in its own HostConfig.LogConfig. lab-web was created before daemon.json existed and still has an empty config, meaning json-file with no rotation. The daemon copies defaults into a container's configuration when the container is created. So changing a default never touches existing containers; you have to recreate them (with Compose, docker compose up -d --force-recreate). Keep a list of what you recreated, or the next disk-full incident will be an old container nobody rolled.

Both user-defined networks came out of 10.200.0.0/16, one /24 each, starting at 10.200.0.0/24. docker0, the default bridge, kept 172.17.0.1/16. That's a consequence of live-restore, not a rule. Without bip, dockerd reuses the address the bridge already has, and with live-restore on it leaves the existing bridge alone. With live-restore off, dockerd deletes and re-creates docker0 at startup. The new one gets the first free subnet from the pools (10.200.0.1/24 with this file). The default bridge's address is also remembered in Docker's network store: remove the pools later and docker0 keeps the pool address. To decide the default bridge's range yourself, set bip (for example "bip": "10.250.0.1/24"), which wins in every case. Pick the pool after you check ip route on the host and the ranges your VPN and cloud networks use. Address clashes don't produce errors. Traffic to the clashing network is routed into the container bridge and disappears.

A valid file that stops dockerd

A common way to expose the API on another socket or on TCP is a hosts key in daemon.json. On a systemd host this breaks the daemon, and the validator doesn't catch it:

--validate saw only the file and said configuration OK. The real start also has the -H fd:// flag from the unit's ExecStart. dockerd refuses to start when an option is set both as a flag and in the file, and the journal names the clash exactly. activating means systemd is retrying under Restart=always, and it gives up after three starts within a minute. Meanwhile ctr -n moby tasks ls shows both containers still running. With live-restore on, a dead daemon is an API outage, not a workload outage. The fix is to reinstall the last known-good file, here merged.json:

reset-failed clears systemd's start-limit counter so the next start is attempted. If you really need another listener, add it with a systemd drop-in that overrides ExecStart. Never put it on plain TCP without TLS client certificates; "The Docker socket and daemon hardening" in Advanced container security covers that.

data-root does not move images on Docker 29

data-root used to move everything Docker stored. With the containerd image store, image content and container snapshots belong to containerd and live under containerd's own root. The containers are removed first. Changing the data directory under running containers is one of the cases where live-restore can't reconnect.

docker info reports root=/srv/docker, yet docker image ls still lists nginx. du shows where image data lives: everything the VM has pulled is in /var/lib/containerd, against a few hundred kilobytes in each Docker directory. The lab- networks are gone, because networks, volumes and container records are dockerd's data and stayed in the old /var/lib/docker. To move images to another disk, change containerd's root in /etc/containerd/config.toml as well. Do it with containers stopped, both daemons stopped and the data copied; "Where Docker keeps data" walks through it. Moving back restores the networks:

storage-driver: overlay2 switches the image store

Older guides put "storage-driver": "overlay2" in every daemon.json. On a fresh Docker 29 host that line doesn't describe the current setup. It replaces it with the legacy graph-driver image store:

docker info now reports overlay2 with the classic graph-driver details. docker image ls finds no nginx, while containerd still holds it. Nothing was deleted; each store hides the other's images. The surprise comes when you remove the line again:

The daemon stays on overlay2. Once /var/lib/docker/overlay2 and /var/lib/docker/image/overlay2 exist, dockerd treats the host like one upgraded from an older engine and keeps the legacy store. That's how upgraded hosts avoid losing their images. The way back is to ask for the containerd store explicitly:

features.containerd-snapshotter: true selects the containerd store regardless of leftover graph-driver state, and nginx is visible again. Set to false, the same flag is how you'd deliberately stay on the legacy store. Set to true on a host upgraded from Docker 28, it's the migration switch. Images and containers in the old store are hidden after it. So docker save the images you need before switching and docker load them after, or pull and rebuild them, then recreate the containers. On a fresh Docker 29 host, leave both storage-driver and the feature flag out.

Roll back

Rolling back means restoring the backup. Here that's the {} recorded at the start. It restores the defaults for everything except the image store, which stays on the legacy store for the reason above:

Deleting the two overlay2 directories is safe here only because the legacy store never held an image on this VM. On a real host those directories are where an upgraded engine keeps its images, so do not copy that step outside the lab.

Keys to leave alone

Keep daemon.json in the same repository as the rest of the host build. Run dockerd --validate on it in CI, and roll changes to one canary host before the rest.

Your turn

On the sec VM, copy typo-key.json to a new file and fix the parent key back to log-opts, but misspell max-size inside it. Before you run sudo dockerd --validate --config-file on it, predict whether the validator rejects it at all, and if so, which key it names. Compare what you get with the two failures above, then delete the copy.

Takeaway

Change daemon.json with a backup, a merge, a validate and a reload before you restart. Make sure live-restore is on first: that's what kept lab-web's shim PID unchanged across the restart. Don't treat configuration OK as proof the daemon will start. The hosts key passed the validator and still stopped dockerd because of -H fd:// in the unit, which is why a canary host belongs in the rollout after the CI check.

Related