Container networking across multiple hosts presents unique challenges. Containers need to communicate with each other regardless of which physical host they're running on, and they need to do so securely and efficiently. Docker Swarm solves these challenges through overlay networks—virtual networks that span across all nodes in your cluster. This article explores how overlay networks work, the VXLAN technology that powers them, and the routing mesh that provides seamless load balancing.

Understanding Overlay Networks

What Are Overlay Networks?

An overlay network is a virtual network layer built on top of an existing physical network infrastructure. In Swarm, overlay networks enable containers running on different hosts to communicate as if they were on the same local network, regardless of the underlying network topology.

When you create an overlay network in Swarm, all nodes participating in services attached to that network can communicate through it. The network abstracts away the complexity of the physical network, presenting a flat, simple networking model to your containers.

Why Overlay Networks Matter

Traditional bridge networks work well on a single host but don't extend across multiple hosts. Overlay networks solve this limitation by creating a distributed virtual network that:

Spans all nodes in the Swarm cluster
Provides automatic service discovery
Enables secure, encrypted communication
Isolates different applications on the same infrastructure
Simplifies network configuration for distributed applications

VXLAN: The Technology Behind Overlay Networks

What is VXLAN?

VXLAN (Virtual Extensible LAN) is the encapsulation protocol that makes overlay networks possible. It creates Layer 2 (Ethernet) segments across Layer 3 (IP) networks by encapsulating Ethernet frames inside UDP packets.

VXLAN was designed to address the scalability limitations of traditional VLANs, which are limited to 4096 network segments. VXLAN supports up to 16 million network segments, making it ideal for large-scale container deployments.

How VXLAN Encapsulation Works

When a container sends a packet to another container on the same overlay network but different host:

The source container creates a standard Ethernet frame
Docker intercepts the frame before it reaches the physical network
The frame is encapsulated inside a UDP packet with a VXLAN header
The UDP packet is sent across the physical network to the destination host
The destination host receives the UDP packet and extracts the original Ethernet frame
The frame is delivered to the destination container

This encapsulation happens transparently—containers are unaware that their traffic is being tunneled across the physical network.

VXLAN Header Structure

The VXLAN header adds approximately 50 bytes of overhead to each packet:

Outer IP header (20 bytes)
Outer UDP header (8 bytes)
VXLAN header (8 bytes)
Original Ethernet frame

The VXLAN header includes a 24-bit VXLAN Network Identifier (VNI), which uniquely identifies each overlay network. Each overlay network you create gets assigned a unique VNI.

VXLAN Network Identifier (VNI)

The VNI serves as the network segment identifier in VXLAN. When Docker creates an overlay network, it automatically assigns a VNI from the available range. This VNI is included in every packet on that network, ensuring traffic is correctly routed to containers on the same overlay network.

You can inspect a network to see its VNI:

docker network inspect my-overlay-network

Look for the "com.docker.network.driver.overlay.vxlanid_list" field in the output.

Creating and Using Overlay Networks

Creating an Overlay Network

Create a basic overlay network:

docker network create \
  --driver overlay \
  my-overlay

This creates an overlay network that spans all nodes in the cluster. Services attached to this network can communicate with each other regardless of which nodes they're running on.

Creating an Attachable Overlay Network

By default, only services can attach to overlay networks. To allow standalone containers to connect:

docker network create \
  --driver overlay \
  --attachable \
  my-attachable-overlay

Now you can attach both services and standalone containers to this network.

Overlay Network with Custom Subnet

Specify a custom IP address range:

docker network create \
  --driver overlay \
  --subnet 10.0.9.0/24 \
  --gateway 10.0.9.1 \
  custom-subnet-overlay

This gives you control over the IP addressing scheme used within the overlay network.

Creating Encrypted Overlay Networks

Enable encryption for traffic on the overlay network:

docker network create \
  --driver overlay \
  --opt encrypted \
  secure-overlay

When encryption is enabled, VXLAN traffic is encrypted using IPSec. This adds security but introduces performance overhead due to encryption/decryption processing.

Overlay Network Scope

Swarm-Scoped Networks

Overlay networks are swarm-scoped by default, meaning they exist across the entire cluster and are managed by the Swarm control plane. Information about overlay networks is stored in the distributed cluster state and automatically propagated to all nodes.

When you create an overlay network on any manager node, it becomes available on all nodes in the cluster.

Network Visibility

A node only creates the actual overlay network interface when a container on that node needs to use it. Before that, the network definition exists in the cluster state but isn't instantiated on the node.

Once a service with tasks on a node connects to an overlay network, Docker creates the necessary network interfaces on that node:

A bridge device for the overlay network
A VXLAN tunnel endpoint (VTEP) device
Namespace-specific virtual ethernet pairs for each container

Service Discovery on Overlay Networks

Built-In DNS Resolution

Overlay networks include embedded DNS resolution. Containers on the same overlay network can reach each other using service names:

# Create a service
docker service create \
  --name web \
  --network my-overlay \
  nginx

# Another service can reach it by name
docker service create \
  --name app \
  --network my-overlay \
  myapp

Inside the app service containers, you can connect to web using the hostname "web". Docker's embedded DNS server resolves this to the IP addresses of all tasks in the web service.

DNS Round-Robin

When a service has multiple replicas, DNS queries return all IP addresses in round-robin fashion. Your application can then connect to any of the returned IPs:

docker service create \
  --name api \
  --network my-overlay \
  --replicas 3 \
  api-image

A DNS query for "api" returns three IP addresses, one for each replica.

VIP-Based Load Balancing

By default, services use a Virtual IP (VIP) for load balancing. When you query a service name via DNS, you get back a single virtual IP address:

docker service create \
  --name backend \
  --network my-overlay \
  --replicas 5 \
  backend-image

DNS queries for "backend" return one VIP. When you connect to this VIP, Docker's built-in load balancer distributes connections across all backend replicas using a round-robin algorithm.

DNS Round-Robin Mode

You can disable VIP and use DNS round-robin instead:

docker service create \
  --name cache \
  --network my-overlay \
  --endpoint-mode dnsrr \
  --replicas 3 \
  redis

DNS queries for "cache" return all three IP addresses directly. Your client application must implement its own load balancing logic.

The Routing Mesh

What is the Routing Mesh?

The routing mesh is Swarm's built-in load balancing mechanism. It ensures that requests to a published service port are automatically routed to an available task, regardless of which node receives the request.

The routing mesh means you can access any service on any node in the cluster, even if that node isn't running a task for that service.

How the Routing Mesh Works

When you publish a service port:

docker service create \
  --name web \
  --publish 8080:80 \
  --replicas 3 \
  nginx

Several things happen:

Port 8080 opens on every node in the cluster
A request to port 8080 on any node is accepted
The routing mesh forwards the request to one of the three web service replicas
The replica can be on any node, including a different node than where the request arrived

This creates a cluster-wide virtual load balancer.

Routing Mesh Architecture

The routing mesh consists of several components:

Ingress Network: A special overlay network created automatically when you initialize a Swarm. All services with published ports are connected to the ingress network.

IPVS (IP Virtual Server): Linux kernel module that provides Layer 4 load balancing. The routing mesh uses IPVS to distribute incoming connections across available tasks.

Network Namespace: Each node maintains a special network namespace for ingress traffic handling.

Ingress Network Deep Dive

The ingress network is a built-in overlay network with special properties:

docker network inspect ingress

Key characteristics of the ingress network:

Created automatically during Swarm initialization
Uses VXLAN for cross-node communication
Includes built-in load balancing via IPVS
Cannot be deleted (though it can be removed and recreated with custom settings)
All services with published ports connect to it automatically

Load Balancing Algorithm

The routing mesh uses IPVS to load balance requests. The default algorithm is round-robin, distributing requests sequentially across all available tasks.

IPVS maintains a connection tracking table, ensuring that connections are routed to the same backend task for their duration. New connections are distributed to different tasks.

Routing Mesh and Task Health

The routing mesh respects task health. If a task fails health checks or becomes unavailable, it's automatically removed from the load balancer pool. New connections won't be routed to unhealthy tasks.

When a task recovers and passes health checks, it's automatically added back to the pool.

Publishing Ports with the Routing Mesh

Host Mode Publishing

Bypass the routing mesh with host mode:

docker service create \
  --name web \
  --publish published=8080,target=80,mode=host \
  nginx

In host mode:

The service port is published only on nodes running tasks
You must connect directly to a node running a task
No automatic load balancing across nodes
Useful when you need direct access to specific task instances

Ingress Mode Publishing (Default)

The default ingress mode uses the routing mesh:

docker service create \
  --name web \
  --publish 8080:80 \
  nginx

This is equivalent to:

docker service create \
  --name web \
  --publish published=8080,target=80,mode=ingress \
  nginx

In ingress mode:

Port published on all nodes
Requests automatically load balanced
Access the service through any node
Ideal for most web services and APIs

Publishing Multiple Ports

Publish multiple ports for a single service:

docker service create \
  --name web \
  --publish 8080:80 \
  --publish 8443:443 \
  nginx

Both ports are handled by the routing mesh and available on all nodes.

Publishing UDP Ports

The routing mesh supports both TCP and UDP:

docker service create \
  --name dns \
  --publish published=53,target=53,protocol=udp \
  dns-server

UDP traffic is load balanced using the same routing mesh mechanism as TCP.

Overlay Network Traffic Flow

Container-to-Container Communication

When two containers on the same overlay network communicate:

Same Host Communication:

Source container sends packet
Packet travels through the overlay bridge device
Packet delivered directly to destination container
No VXLAN encapsulation needed

Cross-Host Communication:

Source container sends packet
Packet enters overlay bridge device
Docker determines the destination is on a different host
Packet is VXLAN-encapsulated
Encapsulated packet sent over physical network
Destination host receives and de-encapsulates packet
Packet delivered to destination container

External Traffic Ingress

When external traffic arrives at a published service port:

Traffic hits any node on the published port
Node's iptables rules capture the traffic
Traffic enters the ingress network namespace
IPVS load balancer selects a target task
If task is on local node, traffic routed directly
If task is on remote node, traffic sent via VXLAN tunnel
Target node delivers traffic to the appropriate container

Return Traffic Path

Response traffic follows the reverse path:

Container sends response
Response travels back through ingress network
Response routed to original requesting node
Node forwards response to external client

Connection tracking ensures response traffic follows the established path.

Network Isolation and Segmentation

Multiple Overlay Networks

Create multiple overlay networks for different purposes:

docker network create --driver overlay frontend-net
docker network create --driver overlay backend-net
docker network create --driver overlay data-net

Services on different overlay networks are isolated from each other:

# Web tier on frontend network
docker service create \
  --name web \
  --network frontend-net \
  nginx

# API tier on both networks
docker service create \
  --name api \
  --network frontend-net \
  --network backend-net \
  api-image

# Database on backend network only
docker service create \
  --name db \
  --network backend-net \
  postgres

The web service can reach api (both on frontend-net), api can reach db (both on backend-net), but web cannot reach db directly.

Service-to-Service Isolation

Overlay networks provide network segmentation without requiring physical network separation. Different applications can run on the same infrastructure while remaining network-isolated.

Connecting Services to Multiple Networks

A service can connect to multiple overlay networks:

docker service create \
  --name gateway \
  --network public-net \
  --network private-net \
  gateway-image

This service can communicate with services on both networks, acting as a gateway between network segments.

Overlay Network Performance Considerations

VXLAN Overhead

VXLAN encapsulation adds overhead:

Packet Size: Additional 50 bytes per packet reduces effective MTU CPU Usage: Encapsulation and de-encapsulation consume CPU cycles Latency: Minimal additional latency (typically < 1ms)

For most applications, this overhead is negligible. High-throughput, low-latency applications may notice the impact.

MTU Considerations

The Maximum Transmission Unit (MTU) for overlay networks is typically 1450 bytes (compared to 1500 bytes for standard Ethernet). This accounts for the VXLAN encapsulation overhead.

Docker automatically sets the correct MTU for overlay interfaces. However, if your physical network uses jumbo frames (MTU > 1500), you can configure a larger MTU for overlay networks:

docker network create \
  --driver overlay \
  --opt com.docker.network.driver.mtu=1400 \
  my-overlay

Ensure the MTU value accounts for VXLAN overhead while staying within your physical network's MTU limits.

Encryption Performance Impact

Encrypted overlay networks use IPSec, which adds computational overhead. Encryption can reduce throughput by 20-40% depending on hardware and packet sizes.

Use encryption only when security requirements justify the performance cost. For internal clusters behind firewalls, unencrypted overlays may be acceptable.

Inspecting Overlay Networks

Viewing Network Details

Inspect an overlay network:

docker network inspect my-overlay

Key information includes:

Network ID and name
Subnet and gateway configuration
Driver options
Connected containers and services
VNI (VXLAN ID)

Listing Connected Services

See which services are using a network:

docker network inspect my-overlay \
  --format '{{range .Services}}{{.Name}} {{end}}'

Network Labels

Add custom labels to networks for organization:

docker network create \
  --driver overlay \
  --label environment=production \
  --label team=platform \
  prod-overlay

Labels help with network management and documentation.

Troubleshooting Overlay Networks

Verifying Network Creation

After creating an overlay network, verify it exists:

docker network ls --filter driver=overlay

Checking Container Connectivity

Test connectivity between containers on an overlay network:

# Get shell in first container
docker exec -it container1 sh

# Ping second container by name
ping container2

# Test service port
curl http://servicename:port

Inspecting Network Namespaces

On a Swarm node, view network namespaces:

ip netns list

Overlay networks create namespaces with IDs matching their network IDs. Inspect namespace details:

ip netns exec <namespace-id> ip addr show

Verifying VXLAN Interfaces

Check for VXLAN interfaces on a node:

ip link show type vxlan

Each active overlay network has a corresponding VXLAN interface.

Common Connectivity Issues

Firewall Blocking VXLAN: Ensure UDP port 4789 (VXLAN default) is open between nodes MTU Mismatch: Verify MTU settings on physical and overlay interfaces Routing Problems: Check that nodes can reach each other on the physical network Stale Network State: Remove and recreate problematic networks

Customizing the Ingress Network

Removing the Default Ingress Network

You can remove and recreate the ingress network with custom settings:

# Remove all services using ingress ports first
docker service ls

# Remove ingress network
docker network rm ingress

You'll be prompted to confirm, as this removes routing mesh capability.

Creating Custom Ingress Network

Create a new ingress network with custom configuration:

docker network create \
  --driver overlay \
  --ingress \
  --subnet 10.11.0.0/16 \
  --gateway 10.11.0.1 \
  custom-ingress

The --ingress flag designates this as the cluster's ingress network.

When to Customize Ingress

Customize the ingress network when:

The default subnet conflicts with your existing network ranges
You need a larger address space for published services
You want specific gateway configurations

Overlay Network Lifecycle

Network Creation Timing

Overlay networks are created lazily on nodes. A node only creates the actual overlay interface when:

A service with tasks on that node connects to the network, or
A standalone container on that node connects to an attachable network

This lazy creation reduces resource usage on nodes that don't need the network.

Network Cleanup

When no containers or services are using an overlay network on a node, Docker eventually cleans up the network interfaces. However, the network definition remains in the cluster state.

To fully remove an overlay network:

docker network rm my-overlay

This only succeeds if no services are using the network.

Removing Networks with Connected Services

Attempting to remove a network with connected services fails:

Error response from daemon: network my-overlay has active endpoints

Remove all services from the network first, then delete the network.

Advanced Overlay Network Patterns

Multi-Tier Application Networking

Structure networks by application tier:

# Frontend network - publicly accessible
docker network create --driver overlay web-tier

# Application network - middle tier
docker network create --driver overlay app-tier

# Data network - restricted access
docker network create --driver overlay data-tier

# Web service: public access
docker service create \
  --name nginx \
  --network web-tier \
  --publish 80:80 \
  nginx

# API service: bridges web and app tiers
docker service create \
  --name api \
  --network web-tier \
  --network app-tier \
  api-image

# Database: only on data tier
docker service create \
  --name postgres \
  --network data-tier \
  postgres

# Backend service: bridges app and data tiers
docker service create \
  --name backend \
  --network app-tier \
  --network data-tier \
  backend-image

This creates clear network boundaries between application layers.

Service Mesh Pattern

Create a dedicated network for service-to-service communication:

docker network create --driver overlay service-mesh

docker service create \
  --name service-a \
  --network service-mesh \
  service-a-image

docker service create \
  --name service-b \
  --network service-mesh \
  service-b-image

docker service create \
  --name service-c \
  --network service-mesh \
  service-c-image

All services can communicate with each other through DNS, with built-in service discovery.

External Network Access

Services on overlay networks can access external resources by default. The host's network stack handles routing to external destinations.

To restrict external access, use firewall rules on individual nodes.

Overlay Networks and Docker Compose

Defining Overlay Networks in Compose

When using Docker Compose with Swarm mode (docker stack deploy), define overlay networks:

version: '3.8'

services:
  web:
    image: nginx
    networks:
      - frontend
    deploy:
      replicas: 3

  app:
    image: myapp
    networks:
      - frontend
      - backend
    deploy:
      replicas: 5

  db:
    image: postgres
    networks:
      - backend
    deploy:
      replicas: 1

networks:
  frontend:
    driver: overlay
  backend:
    driver: overlay

Deploy with:

docker stack deploy -c docker-compose.yml myapp

External Networks in Compose

Reference existing overlay networks:

networks:
  existing-net:
    external: true
    name: my-overlay

This connects services to a network created outside the compose file.

Best Practices for Overlay Networks

Network Naming Conventions

Use clear, descriptive network names:

# Good names
docker network create --driver overlay prod-frontend
docker network create --driver overlay app-backend-private

# Avoid
docker network create --driver overlay net1
docker network create --driver overlay my-network

Network Segmentation Strategy

Design networks around application boundaries:

Create separate networks for different applications
Use shared networks only when services need to communicate
Implement network-level isolation for security and organization

Minimize Network Count

While networks provide isolation, too many networks increase complexity:

Each network consumes cluster resources
Too many networks make troubleshooting harder
Group related services on shared networks when appropriate

Document Network Purpose

Label networks with their purpose:

docker network create \
  --driver overlay \
  --label app=ecommerce \
  --label tier=frontend \
  --label environment=production \
  ecommerce-frontend

Plan IP Addressing

Choose non-overlapping subnets for overlay networks:

docker network create --driver overlay --subnet 10.0.1.0/24 net1
docker network create --driver overlay --subnet 10.0.2.0/24 net2
docker network create --driver overlay --subnet 10.0.3.0/24 net3

Avoid conflicts with physical network ranges and other overlay networks.

Use Encryption Selectively

Enable encryption only for networks carrying sensitive data:

# Sensitive data network - encrypted
docker network create \
  --driver overlay \
  --opt encrypted \
  payment-network

# Internal service communication - not encrypted
docker network create \
  --driver overlay \
  service-mesh

Balance security needs with performance requirements.

Docker Swarm's overlay networks provide seamless multi-host container communication through VXLAN encapsulation. The routing mesh extends this with cluster-wide load balancing, allowing any node to accept traffic for any service. By understanding VXLAN's operation, the ingress network's role, and the routing mesh's capabilities, you can design network architectures that provide efficient, secure, and scalable communication for distributed containerized applications across your Swarm cluster.

Overlay Networks in Docker Swarm: VXLAN, Routing Mesh, and Ingress