Overlay networks provide the connectivity foundation for distributed applications, enabling containers on different hosts to communicate as if they were on the same local network. Understanding overlay networks is essential for designing secure, scalable multi-host applications.
What are Overlay Networks?
An overlay network is a virtual network built on top of an existing physical network infrastructure. Instead of requiring special physical network configuration, overlay networks use encapsulation to tunnel traffic between hosts over the existing network.
The Overlay Concept
Think of overlay networks as virtual cables connecting your containers, regardless of which physical hosts they run on. These virtual cables are actually tunnels through the physical network, making container networking independent of physical network topology.
This abstraction is powerful: you can design your application's network architecture based on application requirements, not physical infrastructure constraints.
Why Overlay Networks Matter
Without overlay networks, enabling container communication across hosts would require complex physical network configuration: VLAN setup, IP address management, routing configuration, and coordination with network administrators.
Overlay networks eliminate these requirements. You create a network with a single command, and containers can communicate securely across hosts without any physical network changes.
How Overlay Networks Work
Overlay networks use encapsulation to transport container traffic across the physical network:
Encapsulation and Tunneling
When a container sends a packet to another container on the overlay network, the packet is encapsulated—wrapped in another packet with headers appropriate for the physical network.
This outer packet is routed through the physical network to the destination host. Upon arrival, the encapsulation is removed, and the original packet is delivered to the destination container.
The containers are unaware of this encapsulation. They believe they're on a simple local network, communicating directly with each other.
VXLAN Technology
Overlay networks typically use VXLAN (Virtual Extensible LAN) for encapsulation. VXLAN tunnels layer 2 (Ethernet) frames over layer 3 (IP) networks, allowing containers to have layer 2 connectivity even when running on hosts connected only via layer 3 routing.
VXLAN adds a 50-byte overhead to each packet (8-byte VXLAN header, 8-byte UDP header, 20-byte IP header, 14-byte Ethernet header). This overhead is usually negligible but can matter in bandwidth-constrained environments.
Gossip Protocol
The overlay network uses a gossip protocol to discover which containers run on which hosts. When a container needs to send traffic to another container, it queries the gossip network to determine the container's physical location.
This information is cached, so subsequent packets don't require lookups. The cache is updated automatically when containers move or restart.
Creating Overlay Networks
Overlay networks are created explicitly for use with distributed applications:
Basic Network Creation
Creating an overlay network is straightforward:
docker network create --driver overlay mynetwork
This creates an overlay network named "mynetwork" that services can attach to. All containers connected to this network can communicate with each other across hosts.
Network Options
Overlay networks support various configuration options:
Subnet Configuration
By default, networks use automatically assigned subnets. You can specify custom subnets:
docker network create --driver overlay --subnet 10.0.1.0/24 mynetwork
Custom subnets provide predictable addressing and help avoid conflicts with existing networks.
Gateway Configuration
You can specify a custom gateway for the network:
docker network create --driver overlay --subnet 10.0.1.0/24 --gateway 10.0.1.1 mynetwork
The gateway is the router for traffic leaving the overlay network.
Multiple Subnets
Networks can have multiple subnets, supporting dual-stack (IPv4 and IPv6) configurations or multiple IPv4 ranges:
docker network create --driver overlay \ --subnet 10.0.1.0/24 \ --subnet 2001:db8:1::/64 \ mynetwork
Attachable Networks
By default, overlay networks are only available to services. Standalone containers cannot connect to them. The --attachable flag changes this:
docker network create --driver overlay --attachable mynetwork
Attachable networks can be used by both services and standalone containers, useful for debugging or one-off tasks that need to communicate with services.
Network Scope and Isolation
Overlay networks provide network isolation between applications:
Network Boundaries
Containers on different overlay networks cannot communicate with each other, even if they're on the same host. Each overlay network is an isolated network segment.
This isolation provides security: applications cannot interfere with each other's network traffic, and accidental misconfiguration in one application won't affect others.
Connecting Multiple Networks
Services can connect to multiple overlay networks simultaneously. A service attached to both "frontend" and "backend" networks can communicate with services on either network.
This enables network segmentation patterns: web services might connect to both a public-facing network and a private backend network, while databases connect only to the private network.
Network Segmentation Example
# Create separate networks for different tiers docker network create --driver overlay frontend docker network create --driver overlay backend docker network create --driver overlay database # Web service connects to frontend and backend docker service create --name web \ --network frontend \ --network backend \ nginx # API service connects to backend and database docker service create --name api \ --network backend \ --network database \ myapi # Database connects only to database network docker service create --name db \ --network database \ postgres
This segmentation ensures the database is not directly accessible from the frontend network, improving security.
Service Discovery on Overlay Networks
Overlay networks include built-in service discovery:
DNS-Based Discovery
Services on overlay networks can find each other using DNS. Every service has a DNS name equal to its service name.
For example, a service named "api" can be reached at "api" from any container on the same network. The DNS server built into the network resolves this name to the service's virtual IP address.
Automatic Updates
When services scale or tasks restart, DNS records are automatically updated. Applications don't need to track individual container IP addresses—they just use service names.
Task Discovery
In addition to service-level DNS, you can access individual tasks using a special DNS format: tasks.<service-name>. This returns all IP addresses of all tasks for a service.
This is useful when you need to know about all instances, perhaps for cache coordination or cluster membership.
Overlay Network Performance
Overlay networks introduce some overhead, but it's usually minimal:
Encapsulation Overhead
VXLAN adds 50 bytes to each packet. For large packets (like 1500-byte Ethernet frames), this is about 3% overhead. For small packets, the percentage is higher.
Most applications send reasonably sized packets, so encapsulation overhead is negligible. However, applications sending many tiny packets might experience measurable overhead.
Encryption Overhead
Overlay networks can be encrypted, adding cryptographic overhead. Encrypted networks use IPsec tunnels to protect traffic between hosts.
Encryption consumes CPU cycles and adds latency (typically a few milliseconds). The security benefits usually outweigh the performance cost, but for extremely latency-sensitive applications, unencrypted networks might be preferable.
Latency Impact
Overlay networks add minimal latency—typically less than 1ms for local traffic. Cross-datacenter or cross-region traffic introduces additional latency based on physical distance, not overlay network overhead.
Network Encryption
Overlay networks support encryption to protect traffic between hosts:
Enabling Encryption
Create an encrypted overlay network with the --opt encrypted flag:
docker network create --driver overlay --opt encrypted secure-network
Traffic between hosts on this network is encrypted using IPsec. Traffic between containers on the same host is not encrypted—encryption only protects multi-host traffic.
Encryption Performance
Encryption uses AES-GCM, a fast authenticated encryption algorithm. On modern CPUs with AES hardware acceleration, encryption overhead is minimal—often less than 5% throughput reduction.
On older CPUs without hardware acceleration, encryption can significantly reduce throughput. Measure performance in your environment if encryption is critical.
When to Encrypt
Use encryption when network traffic traverses untrusted networks or when security policies require encryption. For traffic within a trusted datacenter, encryption might be unnecessary overhead.
Network Drivers
While the overlay driver is standard, understanding driver concepts helps:
Overlay Driver
The overlay driver creates multi-host networks using VXLAN encapsulation. It's the driver you'll use for most distributed applications.
Bridge Driver
The bridge driver creates single-host networks. It's not useful for multi-host applications but is the default for standalone containers.
Host Driver
The host driver removes network isolation, placing containers directly on the host's network. This provides maximum performance but eliminates the isolation and portability benefits of network namespaces.
None Driver
The none driver disables networking entirely. Useful for containers that don't need network access.
Network IP Address Management (IPAM)
The overlay driver includes IP address management:
Automatic IP Assignment
By default, containers on overlay networks receive IP addresses automatically from the network's subnet. The IPAM system ensures no address conflicts.
Fixed IP Addresses
You can assign fixed IP addresses to services:
docker service create --name web \ --network mynetwork \ --endpoint-mode vip \ --endpoint-mode dnsrr \ nginx
Fixed addresses are rarely necessary—service discovery via DNS names is more flexible—but they're useful for specific integration requirements.
IPv6 Support
Overlay networks support IPv6. Create a network with an IPv6 subnet:
docker network create --driver overlay \ --ipv6 \ --subnet 2001:db8::/64 \ mynetwork
Services on this network receive both IPv4 and IPv6 addresses.
Container Connectivity Patterns
Overlay networks enable various connectivity patterns:
Full Mesh Connectivity
By default, all containers on an overlay network can communicate with all other containers. This full mesh connectivity simplifies application design—any component can reach any other component.
Segmented Connectivity
Using multiple networks, you can create segmented connectivity: some services on network A can reach services on network B by joining both networks, while services only on A or only on B remain isolated.
Hub-and-Spoke Patterns
Central services (like message queues or databases) can connect to multiple application-specific networks, acting as hubs that multiple applications share.
Network Lifecycle
Understanding network lifecycle helps manage networks effectively:
Creating Networks
Networks are typically created before deploying services that use them. Pre-creating networks ensures they're available when services start.
Network Persistence
Networks persist until explicitly removed. They survive service removal, allowing services to be stopped and restarted without recreating networks.
Removing Networks
Networks can only be removed when no containers or services are connected to them. Disconnect all services first, then remove the network:
docker network rm mynetwork
Overlay Network Troubleshooting
When networking issues occur, systematic troubleshooting identifies causes:
Connectivity Problems
If containers cannot communicate, verify they're on the same network. Use docker network inspect to see which services are connected.
Check that the overlay network exists and is healthy. Ensure hosts can communicate over the physical network—overlay networks require working underlay connectivity.
DNS Resolution Issues
If DNS names don't resolve, verify the service name is correct and that services are on the same network. Check that the built-in DNS resolver is functioning.
Performance Issues
If network performance is poor, check for network congestion on the physical network. Verify that VXLAN overhead isn't overwhelming your network capacity.
For encrypted networks experiencing performance problems, verify that CPU resources are sufficient for encryption operations.
Network Best Practices
Plan Network Architecture
Design your network architecture before deploying services. Decide which services need to communicate and create appropriate network segmentation.
Use Descriptive Names
Name networks descriptively: "frontend", "backend", "database-private", "monitoring". Clear names make network architecture self-documenting.
Minimize Network Span
Don't connect services to networks they don't need. Limiting network connections reduces attack surface and makes architecture clearer.
Consider Encryption Needs
Evaluate whether encryption is necessary for each network based on security requirements and performance constraints.
Document Network Layout
Document your network architecture: which services connect to which networks and why. This documentation helps with troubleshooting and onboarding.
Advanced Network Features
Network Plugins
Third-party network plugins provide additional capabilities beyond the built-in overlay driver. These plugins might offer advanced routing, integration with external network infrastructure, or specialized networking features.
External Networks
Networks can be marked as external, indicating they were created outside the current deployment and should not be managed automatically:
networks:
existing-network:
external: true
This is useful when multiple deployments share networks or when network creation is managed separately from service deployment.
Network Labels
Labels provide metadata for networks, useful for organization and automation:
docker network create --driver overlay \ --label environment=production \ --label project=myapp \ mynetwork
Network Monitoring
Monitoring network health and performance is essential:
Network Metrics
Track metrics like throughput, packet loss, and latency across overlay networks. These metrics help identify network bottlenecks and capacity issues.
Connection Tracking
Monitor active connections on networks. Excessive connection counts might indicate connection leaks or need for connection pooling.
Error Rates
Track network errors: dropped packets, failed connections, DNS resolution failures. High error rates indicate network problems requiring investigation.
Network Security Considerations
Overlay networks have security implications:
Network Isolation
Use overlay networks to isolate sensitive services. Don't connect untrusted services to networks containing sensitive data.
Encryption for Sensitive Data
Encrypt networks carrying sensitive data, especially if hosts are in different security zones or datacenters.
Access Control
While overlay networks provide isolation, they don't provide authentication. Services on the same network can communicate freely. Implement application-level authentication for sensitive services.
Network Policies
Consider using network policy features (if available in your environment) to further restrict traffic even within overlay networks, allowing only necessary communication.
Overlay Networks and Physical Network
Overlay networks interact with the physical network:
Underlay Requirements
Overlay networks require working IP connectivity between hosts. Hosts must be able to reach each other over UDP port 4789 (VXLAN) and TCP/UDP port 7946 (gossip protocol).
Firewalls must allow this traffic, or overlay networks cannot function.
MTU Considerations
Overlay encapsulation adds 50 bytes to packets. If your physical network has a standard 1500-byte MTU, overlay packets can be up to 1550 bytes, potentially causing fragmentation.
Consider reducing the overlay network MTU or increasing the physical network MTU to accommodate encapsulation without fragmentation.
Physical Network Performance
Overlay network performance is limited by physical network performance. A 1 Gbps physical network cannot support aggregate overlay traffic exceeding 1 Gbps (minus encapsulation overhead).
Multi-Datacenter Considerations
Overlay networks can span datacenters, but with considerations:
Latency Impact
Cross-datacenter latency affects overlay network performance. If physical latency is 50ms, overlay latency will be at least 50ms.
Design applications to tolerate this latency or avoid cross-datacenter communication where possible.
Bandwidth Costs
Traffic between datacenters often incurs bandwidth charges. Overlay networks don't reduce these costs—they make cross-datacenter communication easier, but you still pay for bandwidth used.
Failure Domains
Cross-datacenter overlay networks span failure domains. If datacenter connectivity fails, services in different datacenters lose communication, potentially causing service failures.
Design applications to handle partitions gracefully or avoid cross-datacenter dependencies.
Network Design Patterns
Three-Tier Network
Create separate networks for web, application, and database tiers. Web services connect to the web and application networks. Application services connect to application and database networks. Databases connect only to the database network.
This pattern limits exposure: databases are not directly accessible from the web tier.
Shared Services Network
Create a network for shared infrastructure services (monitoring, logging, message queues). All application networks connect to the shared services network, enabling applications to use shared infrastructure without complex network configuration.
Per-Application Networks
Give each application its own network. Applications are completely isolated from each other. Shared services that multiple applications use join multiple application networks.
This pattern maximizes isolation at the cost of more complex network management.
Network Planning Guidelines
Subnet Sizing
Choose subnet sizes appropriate for your expected container counts. A /24 subnet provides 254 addresses, suitable for small deployments. Larger deployments might need /16 or larger subnets.
Avoid Overlaps
Ensure overlay network subnets don't overlap with physical network subnets or other overlay networks. Overlaps cause routing ambiguity and connectivity problems.
Reserve Ranges
If you might create many networks, plan a range of subnets you'll use for overlay networks. This prevents accidentally using subnets that conflict with future needs.
Network Performance Optimization
Reduce Encapsulation Overhead
For maximum performance, use jumbo frames on your physical network (MTU > 1500). This reduces the percentage overhead of VXLAN encapsulation.
Consider Encryption Trade-offs
Encryption provides security but reduces performance. For internal trusted networks where security policies don't require encryption, use unencrypted overlays for better performance.
Optimize Application Protocols
Use efficient application protocols that minimize per-packet overhead. Since each packet incurs encapsulation overhead, reducing packet count improves efficiency.
Co-locate When Possible
When possible, schedule services that communicate heavily onto the same hosts. This avoids network encapsulation and improves performance.
Overlay networks are the foundation of multi-host container networking. They provide secure, isolated networks that enable containers to communicate across hosts without complex physical network configuration. Understanding their operation, capabilities, and best practices enables you to build robust, scalable networked applications.