Overlay networks and the routing mesh

VXLAN overlays, VIP vs DNS round robin, the ingress mesh, host mode and encryption.

Every node in the swarm reports Ready, and DNS resolves the service name. Yet a request from a container on one node to a task on another still times out. Managers and workers talk over TCP 2377 and gossip over 7946, so the cluster looks healthy. Service traffic between nodes takes a different path, VXLAN over UDP port 4789. A single firewall rule on that port breaks every cross-node connection and leaves the control plane untouched. You'll follow that path here: overlay networks, service discovery, the routing mesh, encrypted overlays, and the firewall requirements that go with them.

The lab is the simulated cluster from "Swarm clusters and services", started from ~/lab/swarmnet where the lesson files unpack. It has three docker:29-dind nodes (mgr1 10.77.0.11, wrk1 10.77.0.12, wrk2 10.77.0.13) on the VM bridge lab-swarm, reached through Docker contexts. swarm-lab.sh builds them, and it pulls any base image the VM lacks before copying it into the nodes. The nodes share the VM's kernel and bridge, so the VM can capture their traffic and filter it. That's how you'll reproduce the timeout further down. Between real hosts the same packets cross a physical network and its firewalls.

The last command builds the small nginx test image from that lesson and pushes it to the lab registry. It answers with its version, its container hostname and the client address it sees.

An overlay network

An overlay network is a layer-2 network for containers that spans hosts. Each node attached to it gets a Linux bridge for the network inside a dedicated network namespace, plus a VXLAN interface. A frame from a container on one node is wrapped in a UDP packet and sent to the node that hosts the destination container. It's unwrapped there. You create overlays on a manager, and you can make them attachable. --attachable lets standalone containers (docker run) join, which is handy for debugging.

Before any task used appnet, wrk2 didn't know about it. Managers create overlays in the cluster state and extend them to a worker only when a task on that worker needs the network. ingress is the overlay Swarm creates for the routing mesh. appnet got the subnet 10.0.1.0/24 from Swarm's default pool and VXLAN network identifier 4097. The default pool is 10.0.0.0/8 carved into /24s; change it with docker swarm init --default-addr-pool if it collides with your networks.

Names: a VIP, tasks., or DNS round robin

Docker's embedded DNS server (127.0.0.11 in every container) answers for services on networks the container shares with them. A debug container on appnet asks for api and for tasks.api:

api resolves to one address, 10.0.1.2, the service's virtual IP (VIP). tasks.api returns the two task addresses. Connections to the VIP are balanced across healthy tasks by IPVS in the kernel of the calling node. That's why four requests alternated between the two containers. The VIP stays the same while tasks are replaced, rescheduled or scaled, so clients that cache DNS answers keep working. Balancing is per connection, not per request: a client that holds one keep-alive connection open talks to one task.

With --endpoint-mode dnsrr there's no VIP (VirtualIPs is null), and the service name itself returns every task address. Clients then do their own balancing, and DNS caching becomes your problem. A client that resolved once and kept the answer keeps calling a task that may have moved. Use dnsrr for software that wants to see its peers (clustered databases, some proxies with their own health checking), and the default VIP mode otherwise. A dnsrr service can't publish ports through the routing mesh. It can publish in host mode, shown below.

VXLAN on the wire

Now capture on the VM bridge while the debug container on mgr1 opens a connection to an api task:

Each capture line is two packets. The outer one is the UDP packet from node to node: 10.77.0.11 to 10.77.0.12, destination port 4789, VXLAN network identifier 4097 (the ID appnet showed above). Inside it is the container-to-container TCP SYN from the debug container (10.0.1.8) to the api task at 10.0.1.4, port 80, and then the SYN-ACK coming back. The connection went to the VIP, but IPVS on mgr1 had already rewritten the destination to a task address before encapsulation. The inner traffic isn't encrypted. Anyone who can capture on the network between the hosts reads it in plain text. That matters as soon as that network is shared or crosses a data centre you don't control.

Encapsulation costs you 50 bytes per packet. The inner MSS of 1410 above comes from an inner MTU of 1450 on a 1500-byte network. Some VPNs and cloud networks have a smaller underlying MTU. There, set the overlay MTU explicitly with docker network create -d overlay --opt com.docker.network.driver.mtu=<value>, or small requests work and large responses hang.

A firewall that drops 4789/udp

Now the opening symptom. A rule in the VM's DOCKER-USER chain drops UDP port 4789 on the bridge between the nodes. Between real hosts, a host firewall or a security group without that port does the same. Node status and DNS still work, and the TCP connection to the VIP times out:

Delete the rule and the same request goes through:

You'll see the same picture when a cloud security group allows 2377 but not 4789/udp. It also appears when 7946 is blocked (node discovery then flaps and overlay peers aren't learned), or when a host firewall on one node filters forwarded traffic. Between all nodes, allow 7946/tcp, 7946/udp and 4789/udp. Towards managers, allow 2377/tcp, and IP protocol 50 (ESP) when you use encrypted overlays. Keep all of them closed to everything outside the cluster. VXLAN has no authentication, so anyone who can send packets to 4789 on a node can inject traffic into its overlays.

The routing mesh

A port published with -p goes on the ingress overlay. Every node in the swarm listens on it, whether or not it runs a task, and forwards the connection to a healthy task on any node:

The single web task runs on wrk1, yet all three node addresses answer, and the reply comes from the same container each time. The client address the application sees is 10.0.0.2, 10.0.0.3 or 10.0.0.4. That's the ingress-network address of the node that received the connection, not the VM's address. The mesh applies source NAT so that replies return through the node that accepted the connection. It's convenient behind an external load balancer that knows nothing about task placement. But published ports are open on every node, and your application loses the real client address. That breaks IP allow-lists, rate limits and access logs.

Host-mode publishing binds the port only on nodes that run a task. It goes directly to the task, without the mesh and without NAT:

mgr1 runs no edge task and refuses the connection (curl exit 7). The two workers answer with the client address 10.77.0.1, the VM's own address on the bridge, which is the real source. docker service ls lists only mesh ports in its PORTS column, so host-mode ports don't show up there; check the service spec instead. Host mode pairs naturally with global services: one task per node, and each node publishes the port. Then a load balancer that health-checks the nodes sends traffic to them, or a proxy such as Traefik or HAProxy runs as that global service. With a replicated service in host mode, two tasks of the same service can't land on one node (the port is taken). And your load balancer has to follow task placement.

Encrypted overlays

--opt encrypted makes Swarm set up IPsec (ESP in transport mode, AES-GCM) between every pair of nodes that share the network. The managers generate the keys and rotate them every 12 hours.

The same kind of request now shows ESP packets between the nodes, with no VXLAN header and no inner addresses visible. Encryption is per network and off by default. It costs CPU and throughput on every packet, so measure it on your hardware before you turn it on for heavy east-west traffic. Firewalls between nodes must pass IP protocol 50, which many security group templates don't include, and the failure looks exactly like the 4789 drop above. Encryption covers overlay traffic only. The routing mesh's ingress network isn't encrypted by this option. Traffic from clients to published ports is whatever the application makes it, so terminate TLS in the application or the proxy as well. On Windows nodes, encrypted overlays aren't supported.

Swarm and the nftables firewall backend

Docker 29 added an experimental nftables firewall backend ("firewall-backend": "nftables" in daemon.json), which "Publishing ports and the packet path" covers. The overlay driver's rules haven't been migrated, so the daemon refuses to enter swarm mode with it. The first command backs up daemon.json (an empty {} when there's none, as here). It merges the key into the backup with jq, validates the result and restarts. The last one restores the backup:

If you moved a host to the nftables backend, it can't be a swarm node until you switch back to iptables, the default. The reverse also holds, since a daemon in swarm mode keeps iptables. Plan the backend per host role.

Try this

On your next run, before down, put the 4789 drop back with the same sudo iptables -I DOCKER-USER command and repeat the routing-mesh loop over the three node addresses. The web task runs only on wrk1. Predict which addresses still answer and which time out, then check, and delete the rule with the fw-allow command afterwards.

Takeaway

A swarm can show every node Ready and resolve api to 10.0.1.2 while service traffic between nodes is dead, because the control plane and VXLAN use different ports. When a cross-node request times out, check UDP 4789 (and IP protocol 50 on encrypted overlays) on every host and security group first. Ports 2377 and 7946 being open says nothing about the data path.

Related