Swarm clusters and services

Managers, Raft quorum, join tokens, services, rolling updates, rollback and autolock.

Here's where you're heading: a three-node cluster, as its only manager reports it. The cluster doesn't exist yet, so there's nothing to run here. The same command comes back once you've built it.

That's Swarm mode: several Docker engines joined into one cluster. A manager (mgr1, the Leader) accepts docker service commands, and two workers run whatever the manager assigns. The ENGINE VERSION column is the ordinary Docker Engine. Swarm is built into it, and there's nothing else to install. You'll build that cluster, run services on it, and update and roll them back. Then you'll take nodes and managers away to see what breaks and what keeps running.

Where Swarm stands

Swarm mode ships in every Docker Engine release and is maintained. Its core is the open-source SwarmKit project, maintained mainly by Mirantis, which bought Docker's enterprise business in 2019. It changes slowly. New orchestration work (autoscaling, operators, admission policy, most of the tooling) happens around Kubernetes, the usual choice for large fleets. Swarm suits a handful of hosts run by a team that knows Docker and Compose. It gives you scheduling, rolling updates, secrets and overlay networks without a separate control plane. For a single host, Docker's documentation points to Compose.

The lab: three nodes in one VM

A real swarm is several machines. The lab simulates them with Docker-in-Docker. Each node is a privileged docker:29-dind container running its own dockerd on the sec VM, with its own IP address on a bridge network lab-swarm (10.77.0.0/24). The script below, in the lesson files, starts the three nodes and a plain registry (lab-reg:5000) they can all pull from. It copies a few base images from the VM into the nodes, so they never pull from Docker Hub themselves. And it creates a Docker context per node from the TLS client certificates the dind image generates. The nodes run with --insecure-registry 10.77.0.0/24, which lets them talk plain HTTP to the lab registries. That's acceptable on a private lab bridge and nowhere else, since a real registry needs TLS.

Run it from ~/lab/swarmintro, where the lesson files unpack:

#!/bin/sh
# Simulates three Docker hosts inside one lab VM: Docker-in-Docker containers lab-mgr1, lab-wrk1 and
# lab-wrk2 on the bridge network lab-swarm (10.77.0.0/24), plus a registry lab-reg:5000 they all use.
# Each node is a full dockerd (docker:29-dind, privileged); --tmpfs /run gives a restarted node an empty
# /run, as after a reboot. The node's TLS client certificates become a Docker context of the same name,
# so `docker --context lab-wrk1 info` talks to that node's daemon. Base images are pulled once on the VM
# (only if missing) and copied into lab-mgr1, so the nodes never pull from Docker Hub; alpine and postgres
# are pushed to lab-reg for the other nodes.
# Usage: ./swarm-lab.sh up|down      (runs the VM's docker through sudo; contexts belong to the caller)
[ -f /etc/secopslog-lab ] || { echo "refusing: run this only in the SecOpsLog lab VM" >&2; exit 2; }
set -eu
NODES="lab-mgr1:10.77.0.11 lab-wrk1:10.77.0.12 lab-wrk2:10.77.0.13"
IMAGES="nginx:1.31-alpine alpine:3.24 postgres:18-alpine"
case "${1:-}" in
up)
  for img in $IMAGES; do
    sudo docker image inspect "$img" >/dev/null 2>&1 || { echo "pulling $img (not on this VM yet)"; sudo docker pull -q "$img" >/dev/null; }
  done
  sudo docker network create --subnet 10.77.0.0/24 lab-swarm >/dev/null
  sudo docker run -d --name lab-reg --network lab-swarm --ip 10.77.0.5 registry:3 >/dev/null
  for n in $NODES; do
    name=${n%%:*}; ip=${n#*:}
    sudo docker run -d --privileged --name "$name" --hostname "${name#lab-}" --tmpfs /run \
      --network lab-swarm --ip "$ip" docker:29-dind --insecure-registry 10.77.0.0/24 >/dev/null
  done
  for n in $NODES; do
    name=${n%%:*}; ip=${n#*:}
    i=0; until sudo docker exec "$name" docker info >/dev/null 2>&1 && sudo docker exec "$name" test -f /certs/client/key.pem; do
      i=$((i+1)); [ $i -lt 60 ] || { echo "$name did not start" >&2; exit 1; }; sleep 1; done
    d=$(mktemp -d)
    sudo docker cp "$name:/certs/client" - | tar -x -C "$d"
    docker context create "$name" --description "lab node ${name#lab-}" \
      --docker "host=tcp://$ip:2376,ca=$d/client/ca.pem,cert=$d/client/cert.pem,key=$d/client/key.pem" >/dev/null
    rm -rf "$d"
    echo "$name  $ip  $(docker --context "$name" version -f '{{.Server.Version}}')  ready"
  done
  for img in $IMAGES; do sudo docker save "$img" | docker --context lab-mgr1 load -q >/dev/null; done
  for img in alpine:3.24 postgres:18-alpine; do
    docker --context lab-mgr1 tag "$img" "lab-reg:5000/$img"
    docker --context lab-mgr1 push -q "lab-reg:5000/$img" >/dev/null
  done
  echo "lab-reg:5000 has: $(curl -s http://10.77.0.5:5000/v2/_catalog)"
  ;;
down)
  docker context use default >/dev/null 2>&1 || true
  for n in $NODES; do docker context rm -f "${n%%:*}" >/dev/null 2>&1 || true; done
  sudo docker rm -f -v lab-mgr1 lab-wrk1 lab-wrk2 lab-reg >/dev/null 2>&1 || true
  sudo docker network rm lab-swarm >/dev/null 2>&1 || true
  echo "swarm lab removed"
  ;;
*) echo "usage: $0 up|down" >&2; exit 2 ;;
esac

This VM had no postgres:18-alpine yet, so the script pulled it first. On a VM that already has all three images, that line doesn't appear. Each node then reports its address and engine version, and the last line lists what the lab registry holds.

A Docker context is a named daemon endpoint plus credentials. docker --context lab-wrk1 info talks to the worker's daemon over TLS on port 2376. docker context use lab-mgr1 makes the manager the default for every later command, so from here on a plain docker command goes to mgr1. You'd use the same mechanism (usually with host=ssh://user@host) to manage a real remote swarm from your workstation.

Here's what the simulation doesn't give you. The three nodes share one kernel, one clock and one physical network. So a kernel panic, a clock skew or a cable failure can't hit one node alone, and there's no firewall between them. On real hosts, open these between the nodes only:

2377/tcp to the managers (cluster management and joins); 7946/tcp and 7946/udp between all nodes (node discovery and gossip); 4789/udp between all nodes (VXLAN overlay traffic); and IP protocol 50 (ESP) if you use encrypted overlays. Keep every one of them closed to everything outside the cluster. 2377 accepts joins from anyone holding a token. VXLAN has no authentication, so anyone who can send packets to 4789 on a node can inject traffic into its overlays. "Overlay networks and the routing mesh" shows that traffic on the wire. The --tmpfs /run in the script makes a restarted node start with an empty /run, as a rebooted host would.

Creating the swarm

docker swarm init turns mgr1 into a one-node swarm and makes it the first manager. --advertise-addr is the address the other nodes will use to reach it. On a host with several interfaces, set it explicitly, or the daemon picks one or refuses to guess. The node ID (25 characters) and the join token are random per cluster, so yours will differ.

A join token has three parts. There's a version (SWMTKN-1), then a digest of the cluster's root CA certificate, which lets the joining node verify it's talking to the right cluster, and then a secret. Worker and manager tokens differ only in the secret. With the worker token and access to port 2377, anyone can add a machine that receives tasks and the secrets granted to them. The manager token adds a machine with a full copy of the cluster state. Tokens don't expire. Rotate them when they may have leaked, or after you finish adding nodes:

After --rotate, the old worker token is rejected with A valid join token is necessary to join this cluster, and only the new one works. Nodes that already joined aren't affected. Rotating the manager token works the same way with docker swarm join-token --rotate manager.

docker node ls at this point is the listing you saw at the top. * marks the node the command ran on, and the AVAILABILITY and MANAGER STATUS columns come back later. A worker refuses cluster commands, because only managers hold the cluster state:

Managers, Raft and quorum

Managers keep the cluster state (nodes, services, networks, secrets, configs) in a replicated log maintained with the Raft consensus algorithm. One manager is the leader and makes changes. A change is committed only when a majority of managers, the quorum, has stored it. With N managers the cluster tolerates the loss of (N-1)/2 of them, rounded down:

Two managers are worse than one. The quorum is still two, so losing either one stops the cluster, and you now have two machines that can fail. Run three managers in production, or five if you must survive two failures or one failure during maintenance. More managers make every write slower. Spread managers across failure domains (racks, availability zones), and keep their clocks synchronised. Keep workloads off them if they're busy (drain them, below).

Services and tasks

You don't start containers on a swarm; you declare services. A service says which image to run, how many copies, with which ports, networks and update policy. The managers keep creating tasks until reality matches. A task is one scheduled unit of work bound to a node, and on that node it becomes one container. Images must come from a registry every node can reach, because each node pulls for itself. The lab builds three versions of a small nginx image on mgr1 and pushes them to lab-reg:5000. Version 2 is a normal release; 3 listens on the wrong port, so its health check fails:

FROM nginx:1.31-alpine
ARG VERSION=1
ARG PORT=80
ENV VERSION=$VERSION PORT=$PORT
COPY default.conf.template /etc/nginx/templates/default.conf.template
HEALTHCHECK --interval=2s --timeout=2s --start-period=2s --retries=2 \
  CMD wget -q -O /dev/null http://127.0.0.1/healthz || exit 1

docker service create prints the service ID, then waits and reports progress. It goes on until the tasks are running and have stayed up for the monitor period (verify: Waiting 10 seconds...), and ends with verify: Service <ID> converged. In a terminal those lines redraw in place as progress bars. The lab captured them without a terminal, so each refresh is a new line (most are elided here). -d returns at once instead, and -q waits silently.

REPLICAS reads running over desired: 3/3 is converged. docker service ps lists the tasks, one per slot (web.1 to web.3), and the scheduler spread them over the three nodes. The service spec now holds the tag plus @sha256: and a digest. At create time the CLI resolved the tag to a digest through the registry and pinned it. So every node runs exactly the same image, even if someone pushes a new web:1 later. When the registry can't be reached, the CLI warns image ... could not be accessed on a registry to record its digest and each node resolves the tag on its own. Treat that warning as an error in a pipeline.

The service publishes port 8080 on every node, including nodes without a task. The client address the app sees is a 10.0.0.x address of Swarm's ingress network rather than the VM's. That's the routing mesh, which "Overlay networks and the routing mesh" takes apart.

A replicated service runs a number of copies wherever they fit. A global service runs exactly one task on every node that's available, including nodes that join later. That suits per-host agents such as log shippers and monitoring:

Global task names end in the node ID rather than a slot number, and there's nothing to scale.

Rolling updates and rollback

The create command above already set the update policy, because that's where it belongs. --update-parallelism 1 replaces one task at a time, and --update-delay 5s waits between batches. --update-monitor 10s watches each new task for 10 seconds after it starts. --update-failure-action rollback reverts the whole service to its previous spec if a task fails during that window. The defaults are parallelism 1, no delay, a 5-second monitor and pause. A failure means a task that exits or becomes unhealthy within the monitor window. --update-max-failure-ratio (default 0) sets how many such failures the update tolerates, as a fraction of the tasks updated, before the failure action fires.

Each slot shows its history: the running web:2 task and, indented with \_, the web:1 task it replaced on the same node. Version 3 is the broken release:

The update started with one slot, web.3 in this run (the order varies). Its web:3 task never became healthy (nginx listens on 8080, the health check probes port 80), so it was stopped. A second web:3 task fared no better, and within the monitor window the update turned into a rollback. The CLI printed rollback: update rolled back due to failure, UpdateStatus says rollback_completed, and all three slots run web:2 again. Only one slot was ever on the bad version, because parallelism was 1. With the default pause action the update would have stopped instead. You'd be left with one slot on the failed version and the service marked paused, until you run docker service rollback web or another update.

Set the update policy when you create the service, or in the stack file, as "Stacks, secrets and configs" does. Flags given to docker service update are part of the new spec, so a rollback reverts them together with the image. Here a service created with the defaults gets the broken image and the rollback policy in one update:

The update rolled back, and with it the failure action went back to the default pause. The next bad release on that service pauses instead of rolling back.

When running stays below desired

docker service ls shows 0/1 while a service is starting, and also when it never will. You'll find the reason in the task list:

The task stays Pending with no suitable node (scheduling constraints not satisfied on 3 nodes): no node carries the label the constraint asks for. The scheduler picks up the label on its next pass. Three seconds after --label-add, the task is running on wrk2. There are other common ERROR values. insufficient resources on N nodes means reservations larger than any node's free CPU or memory. No such image or pull access denied is a registry or credentials problem (see "Stacks, secrets and configs"). task: non-zero exit (1) is a crashing process. Use --no-trunc, because the default output cuts the message.

Node maintenance: pause, drain, demote

With wrk1 paused, scaling to six placed no new task on it but left its existing web.1 alone. Draining wrk2 moved its three web tasks to mgr1 (the only active node left) and shut down the global agent there. Drain stops every task on the node, global ones included, and the scheduler places replicated tasks elsewhere. Drain before you patch or reboot a node, and wait until docker node ps <node> shows nothing running. Do the work, then set it back:

Setting active again doesn't move tasks back. The scheduler only uses the node for new tasks, so rebalance with docker service update --force <service> if you need to.

Losing quorum

Promote both workers so the cluster has three managers. Then freeze two of them with docker pause, which stops every process in a node container at once. It's the lab's closest equivalent of two hosts dropping off the network:

With one of three managers left there's no majority. docker node ls and every change fail with The swarm does not have a leader, and nothing can be scheduled, scaled or updated. The containers already running on mgr1 are untouched: quorum protects the cluster's ability to make decisions, not the workloads. A real outage would look the same. On top of that, tasks on the lost nodes aren't replaced until the managers agree again.

Once the frozen managers answer again, they re-elect a leader and the cluster accepts changes. The second command shows the trap with drain on a manager: mgr1 is Drain and still Leader. Draining a manager stops it from running tasks, and it keeps its vote. Drain one of three managers for maintenance, lose another, and quorum is gone. To take a manager out of the vote, demote it, and keep the count odd. If quorum is lost for good (two of three managers destroyed), the remaining manager can rebuild a one-manager cluster with docker swarm init --force-new-cluster. It keeps services and tasks; then you promote new managers. Make sure the lost managers really are gone, or wipe them before they rejoin. A returning old manager still believes in the old cluster.

Autolock and backups

Raft data on each manager, under /var/lib/docker/swarm, is encrypted, including secrets. By default the key that decrypts it sits on the same disk, so anyone who copies a manager's disk gets everything. Autolock encrypts that key with an unlock key that you hold. A manager that restarts stays locked until someone provides it:

After the restart, mgr1 refuses every cluster command with Swarm is encrypted and needs to be unlocked. docker swarm unlock reads the key from the terminal or, as here, from standard input. The workers may show Unknown for a few seconds afterwards, until their next heartbeat; in this run they were already Ready. The price is operational. An unattended reboot of a manager needs a person or a secret store to unlock it. If every manager is locked and the key is lost, the cluster can't be recovered. Keep the key in a password manager or vault, not next to the cluster, and rotate it with docker swarm unlock-key --rotate.

To back up a manager, stop Docker on it, copy /var/lib/docker/swarm, then start Docker again. (On Docker 29 the swarm state is still under the daemon's data root.) Do it on a non-leader manager so the cluster keeps quorum meanwhile. With autolock on, the copy is useless without the unlock key, so record which key belongs to which backup (stored separately). Restoring means putting that directory on a fresh host and running docker swarm init --force-new-cluster. Do that only when the old managers are gone for good. If any of them comes back, you've got two clusters claiming the same identity and the same workers. The restored state includes the old node list, so remove nodes that no longer exist (docker node rm) and promote new managers. With autolock on, unlock the restored manager with the key that belongs to the backup before anything else. Volume data isn't part of it.

Clean up

./swarm-lab.sh down switches back to the VM's own daemon. It removes the three nodes with everything inside them, the registry and the network. The next two lessons start again from ./swarm-lab.sh up.

Try this

On your next run of the lab, before down, drain wrk1 with docker node update --availability drain wrk1 and predict which nodes docker service ps web --filter desired-state=running will list. Check it, then set wrk1 back to active and run the same listing again. Compare it with the note above: nothing should move back until you run docker service update --force web.

Takeaway

Count managers by votes, not by how many look healthy. Two of three frozen managers were enough for The swarm does not have a leader, and a drained manager still showed Leader. The workloads kept running through it, so a quorum loss in production tends to surface as a failed deploy or scale command rather than as an outage alert.

Related