Load balancing is a fundamental capability that distributes incoming traffic across multiple service instances, ensuring no single instance becomes overwhelmed while maximizing resource utilization and reliability. Understanding how load balancing works enables you to build scalable, resilient applications.
What is Load Balancing?
Load balancing is the process of distributing network traffic across multiple servers or service instances. Instead of routing all requests to a single instance, a load balancer spreads the load, preventing any one instance from becoming a bottleneck.
Why Load Balancing Matters
Without load balancing, clients would need to know about every service instance and manually choose which one to contact. This approach doesn't scale—it's impossible to coordinate clients, manage instance additions or removals, or handle instance failures gracefully.
Load balancing abstracts these concerns. Clients connect to a single, stable endpoint, and the load balancer handles distributing requests to healthy backend instances. This simplifies client implementation while enabling powerful backend flexibility.
Load Balancing Benefits
Load balancing provides multiple advantages:
Scalability: Add more service instances to handle increased load without changing client configurations. The load balancer automatically includes new instances in its distribution pool.
Reliability: If an instance fails, the load balancer routes traffic to remaining healthy instances. Clients experience no disruption beyond the failed requests that were in flight.
Resource Utilization: Distributing load evenly across instances ensures all resources are used effectively rather than some instances sitting idle while others are overloaded.
Maintenance: You can remove instances for maintenance or updates without affecting service availability. The load balancer routes traffic to remaining instances while work is performed.
Built-In Load Balancing
Container orchestration platforms include integrated load balancing capabilities. When you publish a service port, load balancing is automatically configured—no additional setup is required.
Ingress Routing Mesh
The ingress routing mesh is the mechanism that provides automatic load balancing for published service ports. When a service publishes a port, that port becomes available on every host in the cluster, regardless of which hosts are actually running service tasks.
When a client connects to the published port on any host, the routing mesh intercepts the connection and forwards it to one of the service's tasks. This happens transparently—the client is unaware of the redirection.
How the Routing Mesh Works
The routing mesh uses network routing rules to intercept traffic destined for published service ports. When traffic arrives at any host, routing rules redirect it through an internal overlay network to a host running a service task.
The task receiving the traffic processes the request and sends the response back through the same path. From the client's perspective, they connected directly to the initial host and received a response—the internal routing is invisible.
Layer 4 Load Balancing
The built-in load balancing operates at layer 4 of the network stack (the transport layer). This means load balancing decisions are made based on connection information: source IP, source port, destination IP, and destination port.
Layer 4 load balancing is fast and efficient because it doesn't need to inspect packet contents. As soon as a connection is established, it's routed to a backend task, and all traffic for that connection continues to the same task.
Load Balancing Algorithms
The routing mesh uses specific algorithms to decide which service task should receive each connection.
Round Robin Distribution
The default algorithm is round robin: connections are distributed sequentially across available tasks. The first connection goes to task A, the second to task B, the third to task C, then back to task A for the fourth, and so on.
Round robin works well when all tasks have similar capacity and requests have similar resource requirements. It ensures even distribution of connections across all available tasks.
Connection-Level Balancing
Load balancing happens at the connection level, not the request level. This distinction is important for understanding behavior.
Once a connection is established and routed to a specific task, all traffic on that connection continues to the same task until the connection closes. New connections are distributed according to the load balancing algorithm, but existing connections are sticky.
For HTTP/1.1 with keep-alive connections, a single client connection might carry multiple requests, all routed to the same task. For HTTP/2 with multiplexed streams, all streams on a connection reach the same task.
Implications of Connection-Level Balancing
Connection-level balancing means load distribution depends on connection patterns. Many short-lived connections distribute load evenly. Few long-lived connections can result in uneven distribution.
If one client establishes a long-lived connection that sends many requests, those requests all go to the same task, potentially overloading it while other tasks sit idle.
Service Endpoint Access
Published services can be accessed through multiple endpoints, each providing load balancing:
Access via Any Host
You can connect to the published port on any cluster host. The routing mesh ensures your connection reaches a healthy service task regardless of which host you contacted.
This provides flexibility for clients. They can connect to whichever host is most convenient or available, and load balancing still functions correctly.
Virtual IP Access
Services also have a virtual IP address (VIP) assigned within the cluster. Containers can use this VIP to connect to the service, and connections are automatically load-balanced.
The VIP provides a stable network identity for the service. It doesn't change when tasks are created or destroyed, providing consistent connectivity.
DNS-Based Access
Services are resolvable via DNS using their service name. When a container performs a DNS lookup for a service name, it receives the service's virtual IP address.
This allows services to connect to each other using simple, human-readable names rather than IP addresses, while still benefiting from load balancing.
Health-Based Routing
Load balancing only routes traffic to healthy tasks. If a task fails health checks or stops responding, it's automatically removed from the load balancing pool.
Health Check Integration
Services can define health checks that periodically verify task health. When a task fails its health check, the orchestrator marks it unhealthy and stops routing new connections to it.
Existing connections to an unhealthy task are not immediately terminated—they continue until closed by the client or task. However, new connections are directed only to healthy tasks.
Automatic Recovery
When an unhealthy task is replaced with a healthy one, the new task is automatically added to the load balancing pool. This happens without manual intervention or configuration changes.
This self-healing behavior ensures load balancing adapts to task failures and replacements, maintaining service availability.
Load Distribution Patterns
How load is distributed depends on connection patterns and task count:
Even Distribution with Many Connections
When clients make many short-lived connections (common in RESTful HTTP APIs), round robin distribution results in very even load distribution. Each connection is independent, allowing the algorithm to distribute them evenly.
Uneven Distribution with Few Connections
When clients make few long-lived connections (common in WebSocket or gRPC applications), load distribution may be uneven. Some tasks might handle many active connections while others handle few.
This unevenness isn't a failure of load balancing—it's an inherent property of connection-level balancing with persistent connections.
Task Count Impact
With few tasks, even slight unevenness in connection distribution can significantly affect load. With many tasks, minor variations in per-task load have less impact on overall system behavior.
Load Balancing Across Hosts
The routing mesh distributes load not just across tasks but also considers task placement across hosts:
Multi-Host Distribution
When service tasks run on multiple hosts, the routing mesh distributes connections across all tasks regardless of their host. A client connecting to host A might have their connection routed to a task on host B or C.
This cross-host routing ensures all tasks receive traffic, not just tasks on hosts that clients directly contact.
Network Overhead
Cross-host routing introduces network overhead. Traffic enters at one host, traverses the overlay network to another host, and is processed there. Response traffic follows the reverse path.
This overhead is usually minimal but can become significant under very high traffic loads or in network-constrained environments.
External Load Balancers
While built-in load balancing works well for many scenarios, external load balancers provide additional features:
Layer 7 Load Balancing
External layer 7 (application layer) load balancers can inspect request content and make routing decisions based on HTTP headers, URL paths, cookies, or request bodies.
This enables sophisticated routing patterns: sending API requests to one set of tasks and web requests to another, routing based on geographic location indicated in headers, or implementing A/B testing by routing subsets of users to different backends.
SSL Termination
External load balancers can terminate SSL/TLS connections, decrypting traffic before sending it to backend tasks over unencrypted connections within the cluster network.
This offloads cryptographic operations from application tasks and simplifies certificate management by centralizing it in the load balancer.
Advanced Algorithms
External load balancers often support sophisticated load balancing algorithms beyond round robin: least connections, weighted distribution, geographic routing, or custom algorithms based on application-specific metrics.
Session Affinity
External load balancers can provide session affinity (sticky sessions), ensuring all requests from a particular client reach the same backend task.
This is useful for stateful applications that store session state locally in tasks rather than in shared storage.
Publishing Modes
Services can publish ports in different modes, affecting load balancing behavior:
Ingress Mode
Ingress mode is the default. Published ports are available on all hosts, and the routing mesh provides load balancing. This is what we've been discussing throughout this article.
Host Mode
Host mode publishes ports only on hosts actually running service tasks. Each task publishes its port on its host, and no routing mesh is involved.
In host mode, clients must know which hosts run tasks and connect directly to those hosts. Load balancing, if needed, must be implemented externally—perhaps through DNS round robin or an external load balancer.
Host mode is useful when you need predictable port mappings or when the routing mesh overhead is unacceptable, but it sacrifices the convenience and flexibility of automatic load balancing.
Load Balancing Performance
Load balancing performance affects overall application performance:
Latency Impact
The routing mesh adds minimal latency—typically less than a millisecond for local routing. Cross-host routing adds additional latency depending on network topology and distance.
For most applications, this latency is negligible compared to application processing time. However, for extremely latency-sensitive applications, direct routing (avoiding the mesh) might be preferable.
Throughput Impact
The routing mesh can handle high throughput, but it's not infinite. Very high connection rates or data rates might saturate routing mesh capacity.
In practice, application performance usually becomes the bottleneck before routing mesh capacity is exhausted, but this is worth considering for very high-traffic services.
Connection Limits
Each host maintains connection tracking for the routing mesh. Extremely high connection counts (tens of thousands of concurrent connections per host) can exhaust tracking resources.
Most applications never approach these limits, but they're relevant for high-connection-count scenarios like connection pooling aggregators or connection-intensive protocols.
Monitoring Load Distribution
Understanding how load is actually distributed helps optimize application performance:
Task-Level Metrics
Monitor metrics for individual tasks: request counts, response times, resource utilization. Compare across tasks to see if load is evenly distributed.
Significant differences between tasks suggest uneven load distribution. Investigate whether this is due to connection patterns, task health issues, or other factors.
Connection Counts
Track active connection counts per task. This reveals whether connection-level balancing is distributing connections evenly.
If connection counts are even but resource utilization is uneven, different connections might be processing different workloads (some expensive, some cheap).
Response Time Distribution
Monitor response time distributions across tasks. Even load distribution should result in similar response time distributions.
Divergent response times might indicate task health issues, resource constraints, or uneven request complexity distribution.
Troubleshooting Load Balancing Issues
When load balancing doesn't work as expected, systematic troubleshooting identifies the cause:
Connectivity Problems
If clients cannot connect to the service at all, verify that the port is actually published and that hosts are accessible. Check firewall rules and network connectivity.
Uneven Load Distribution
If some tasks are overloaded while others are idle, investigate connection patterns. Are clients using persistent connections? Is the task count appropriate for your traffic patterns?
No Traffic to Specific Tasks
If some tasks receive no traffic, verify they're healthy and registered in the load balancing pool. Check task health check status and logs.
Connection Failures
If connections fail intermittently, unhealthy tasks might still be in the load balancing pool. Verify health checks are configured correctly and that unhealthy tasks are being removed promptly.
Load Balancing Best Practices
Design for Statelessness
Stateless applications work best with connection-level load balancing. Store session state in external systems (databases, caches) rather than locally in tasks.
This allows any task to handle any request, ensuring even load distribution and graceful handling of task failures.
Configure Health Checks
Implement comprehensive health checks that accurately reflect task health. Unhealthy tasks should fail health checks and be removed from load balancing before they affect user experience.
Size Appropriately
Run enough tasks to distribute load effectively. Too few tasks can result in individual tasks becoming overloaded even with perfect load distribution.
Monitor Continuously
Monitor load distribution, connection counts, and task performance continuously. Detect and address imbalances before they affect user experience.
Consider External Load Balancers for Advanced Needs
When built-in load balancing doesn't meet your requirements—you need layer 7 features, session affinity, or advanced routing—integrate external load balancers.
Load Balancing and Scaling
Load balancing and scaling work together:
Load Balancing Enables Scaling
Load balancing makes scaling effective. Adding more tasks without load balancing would require reconfiguring clients to use the new tasks.
With load balancing, scaling up automatically includes new tasks in the distribution pool, immediately increasing capacity.
Scaling Improves Load Distribution
Scaling to more tasks improves load distribution granularity. With more tasks, connection-level balancing can distribute more finely, reducing the impact of long-lived connections.
Dynamic Scaling Considerations
When dynamically scaling based on load metrics, consider that new tasks take time to start and become healthy. Ensure your scaling strategy accounts for task startup time to avoid scaling too slowly.
Advanced Load Balancing Patterns
Geographic Distribution
For globally distributed deployments, route traffic to tasks in regions closest to users. This requires external load balancers with geographic awareness.
Canary Deployments
Route a small percentage of traffic to new versions while most traffic goes to stable versions. This requires external load balancers that can split traffic by percentage.
Blue-Green Deployments
Run two versions of a service (blue and green) and switch all traffic between them. This can be implemented by pointing external load balancers at different service endpoints.
Failover Patterns
Configure primary and backup service deployments. Route all traffic to the primary, but automatically failover to the backup if the primary becomes unavailable.
Protocol Considerations
Different protocols interact with load balancing differently:
HTTP
HTTP requests over persistent connections all reach the same task due to connection-level balancing. HTTP/2's multiplexing amplifies this effect.
Consider using short-lived connections or external layer 7 load balancers for finer-grained balancing.
WebSocket
WebSocket connections are long-lived by design. Once established, all messages traverse the same task. Load balancing happens at connection establishment time.
gRPC
gRPC typically uses long-lived HTTP/2 connections with multiplexed streams. Like WebSocket, effective load balancing happens at connection establishment.
For better load distribution, configure gRPC clients to establish multiple connections or use external load balancers that understand gRPC.
TCP
Raw TCP connections are load-balanced at connection establishment. The connection-level nature is most apparent here—each TCP connection is a distinct unit routed independently.
Security Considerations
Load balancing intersects with security:
TLS Termination
Services can terminate TLS at tasks or at external load balancers. Terminating at tasks provides end-to-end encryption but increases task computational load.
Terminating at external load balancers simplifies certificate management and offloads cryptographic work but means traffic within the cluster is unencrypted.
Source IP Preservation
The routing mesh's network address translation means backend tasks don't see original client IP addresses—they see internal cluster IPs.
If you need client IPs for access control or logging, use host mode publishing or external load balancers that preserve or forward original IPs.
DDoS Protection
The routing mesh provides limited DDoS protection. For serious protection, use external load balancers with sophisticated DDoS mitigation capabilities.
Load Balancing Metrics
Key metrics for understanding load balancing effectiveness:
Requests Per Task
Measure how many requests each task handles. Even distribution means similar counts across tasks.
Connection Count Per Task
Track active connections to each task. This is more relevant than request count for long-lived connection protocols.
Resource Utilization Per Task
Monitor CPU and memory usage across tasks. Even load balancing should result in similar resource utilization.
Error Rate Per Task
Track errors per task. If one task has much higher error rates, it might be unhealthy or overloaded.
Understanding Built-In Limitations
The built-in load balancing has limitations:
Connection-Level Only: No request-level balancing. Long-lived connections can cause uneven load.
Layer 4 Only: No content-based routing or advanced algorithms.
No Session Affinity: Cannot route requests from the same client to the same task.
Limited Metrics: No detailed load balancing metrics like connection latency or failure rates.
For scenarios where these limitations matter, consider external load balancers that provide these capabilities.
Practical Load Balancing Strategy
Effective load balancing strategy combines built-in capabilities with additional tools as needed:
Start with built-in load balancing. For many applications, it's sufficient and requires no additional setup.
Monitor load distribution. Ensure it meets your requirements for performance and reliability.
If built-in load balancing proves insufficient—uneven load distribution, need for advanced features, or protocol-specific requirements—introduce external load balancers.
Design applications to work well with connection-level balancing: use stateless architectures, implement proper health checks, and configure clients appropriately for your protocol.
Load balancing is a fundamental capability that makes distributed applications practical. Understanding how it works, its strengths and limitations, and when to augment it with external tools enables you to build scalable, reliable systems.