Docker Swarm's scheduler determines where to place service tasks across the cluster's worker nodes. While the default scheduling algorithm spreads tasks evenly across available nodes, production workloads often require precise control over task placement. Constraints enforce absolute requirements that must be met for task placement, while preferences express desired characteristics that the scheduler attempts to honor. Understanding these mechanisms enables sophisticated workload placement strategies that optimize for hardware utilization, geographic distribution, failure isolation, and performance requirements.
Understanding the Scheduler's Role
The Swarm scheduler runs on the leader manager and makes all task placement decisions. When a service is created or scaled, the scheduler must decide which nodes should run each task replica.
The scheduler's primary responsibilities include:
- Evaluating which nodes meet a task's requirements
- Selecting optimal nodes from available candidates
- Balancing workload distribution across the cluster
- Respecting user-defined placement rules
- Considering node resources and current utilization
The scheduler makes these decisions for every task placement, whether during initial service creation, scaling operations, or task rescheduling after failures.
Placement Constraints: Absolute Requirements
Constraints define mandatory requirements that nodes must satisfy for task placement. If no nodes meet all constraints, tasks remain in a pending state until suitable nodes become available.
Basic Constraint Syntax
Constraints use a simple expression syntax:
docker service create \ --constraint 'node.role==worker' \ nginx:alpine
This constraint ensures tasks only run on worker nodes, never on managers.
Node Attribute Constraints
Constraints can match various node attributes:
Node role:
--constraint 'node.role==worker' --constraint 'node.role==manager'
Node ID:
--constraint 'node.id==abc123xyz'
Node hostname:
--constraint 'node.hostname==worker-03'
Node platform:
--constraint 'node.platform.os==linux' --constraint 'node.platform.arch==x86_64'
These built-in attributes provide basic placement control based on node characteristics.
Custom Node Labels
Custom labels provide the most flexible constraint mechanism. Administrators assign arbitrary labels to nodes, then reference those labels in service constraints.
Adding Labels to Nodes
docker node update --label-add datacenter=us-east worker-01 docker node update --label-add datacenter=us-west worker-02 docker node update --label-add storage=ssd worker-03 docker node update --label-add storage=hdd worker-04 docker node update --label-add tier=frontend worker-05 docker node update --label-add tier=backend worker-06
Constraining by Custom Labels
# Run only in specific datacenter docker service create \ --constraint 'node.labels.datacenter==us-east' \ myapp:latest # Require SSD storage docker service create \ --constraint 'node.labels.storage==ssd' \ database:latest # Run on backend tier docker service create \ --constraint 'node.labels.tier==backend' \ api:latest
Custom labels enable domain-specific placement logic tailored to your infrastructure.
Constraint Operators
Constraints support multiple comparison operators:
Equality (==):
--constraint 'node.labels.env==production'
Matches nodes where the label equals the specified value.
Inequality (!=):
--constraint 'node.labels.env!=development'
Matches nodes where the label doesn't equal the value, or where the label doesn't exist.
These operators provide both positive and negative matching capabilities.
Multiple Constraints: AND Logic
Multiple constraints create an AND relationship—nodes must satisfy all constraints:
docker service create \ --constraint 'node.role==worker' \ --constraint 'node.labels.datacenter==us-east' \ --constraint 'node.labels.storage==ssd' \ database:latest
This service runs only on worker nodes in the us-east datacenter with SSD storage. All three conditions must be true for task placement.
Node Label Patterns for Placement
Common label patterns for service placement:
Geographic Labels
# Continent/Region/Datacenter hierarchy docker node update --label-add continent=north-america node1 docker node update --label-add region=us-east node1 docker node update --label-add datacenter=us-east-1a node1
# Place service in specific region docker service create \ --constraint 'node.labels.region==us-east' \ geo-distributed-app:latest
Hardware Capability Labels
# CPU and memory tiers docker node update --label-add cpu=high-performance node1 docker node update --label-add memory=high node1 docker node update --label-add gpu=nvidia-v100 node2
# Require GPU-enabled nodes docker service create \ --constraint 'node.labels.gpu==nvidia-v100' \ ml-training:latest
Environment Labels
# Deployment environment docker node update --label-add environment=production node1 docker node update --label-add environment=staging node2 docker node update --label-add environment=development node3
# Production-only services docker service create \ --constraint 'node.labels.environment==production' \ payment-processor:latest
Application Tier Labels
# Network tiers docker node update --label-add tier=dmz node1 docker node update --label-add tier=application node2 docker node update --label-add tier=database node3
# Restrict database to dedicated tier docker service create \ --constraint 'node.labels.tier==database' \ postgres:14
Placement Preferences: Soft Requirements
Unlike constraints, preferences express desired characteristics without making them mandatory. The scheduler attempts to satisfy preferences but will place tasks on nodes that don't match if no better options exist.
Basic Preference Syntax
docker service create \ --placement-pref 'spread=node.labels.datacenter' \ web:latest
This preference asks the scheduler to spread tasks evenly across different datacenter label values.
The Spread Strategy
The spread strategy distributes tasks evenly across nodes with different values for a specified attribute:
# Spread across datacenters docker service create \ --replicas 9 \ --placement-pref 'spread=node.labels.datacenter' \ distributed-cache:latest
If you have three datacenters (us-east, us-west, eu-central) each with three nodes, this creates an even 3-3-3 distribution across datacenters.
How Spread Works
The spread strategy:
- Identifies all unique values for the specified label
- Calculates how many tasks should ideally be on nodes with each value
- Places tasks to achieve the most even distribution possible
If nodes lack the spread label, they form their own group and receive their proportional share of tasks.
Spreading Across Multiple Attributes
Multiple preferences create a hierarchy of spreading:
docker service create \ --replicas 12 \ --placement-pref 'spread=node.labels.datacenter' \ --placement-pref 'spread=node.labels.rack' \ geo-redundant:latest
The scheduler first spreads across datacenters, then within each datacenter, spreads across racks. This creates maximum distribution and fault isolation.
Combining Constraints and Preferences
Constraints and preferences work together—constraints filter eligible nodes, then preferences influence selection among remaining candidates:
docker service create \ --replicas 6 \ --constraint 'node.role==worker' \ --constraint 'node.labels.storage==ssd' \ --placement-pref 'spread=node.labels.datacenter' \ high-performance-db:latest
This service:
- Only considers worker nodes with SSD storage (constraints)
- Among eligible nodes, spreads across datacenters (preference)
Resource-Based Scheduling
The scheduler considers node resources when placing tasks:
Resource Reservations
docker service create \ --reserve-cpu 1.0 \ --reserve-memory 2G \ resource-intensive:latest
The scheduler only places tasks on nodes with at least 1 CPU and 2GB memory available. As tasks are placed, available resources decrease, affecting future scheduling decisions.
Resource Limits
docker service create \ --limit-cpu 2.0 \ --limit-memory 4G \ bounded-app:latest
Limits don't affect scheduling decisions directly but prevent tasks from consuming more than specified resources.
Combining Resources with Constraints
docker service create \ --reserve-cpu 2.0 \ --reserve-memory 4G \ --constraint 'node.labels.tier==compute' \ compute-heavy:latest
This requires nodes in the compute tier with sufficient available resources.
Node Availability States
Node availability affects scheduling:
Active: Node accepts new tasks (default) Pause: Node doesn't accept new tasks but keeps existing tasks running Drain: Node doesn't accept new tasks and existing tasks are rescheduled elsewhere
Set node availability:
docker node update --availability pause worker-01 docker node update --availability drain worker-02 docker node update --availability active worker-03
The scheduler only considers active nodes when placing new tasks.
Label-Based Isolation Patterns
Dedicated Nodes for Specific Services
# Label nodes for dedicated use docker node update --label-add dedicated=postgres node1 docker node update --label-add dedicated=postgres node2 # Service uses dedicated nodes docker service create \ --constraint 'node.labels.dedicated==postgres' \ postgres:14 # Other services avoid these nodes docker service create \ --constraint 'node.labels.dedicated!=postgres' \ general-app:latest
This creates dedicated capacity for critical services.
Multi-Tenancy Isolation
# Label nodes by tenant docker node update --label-add tenant=acme-corp node1 docker node update --label-add tenant=globex node2 # Tenant-specific services docker service create \ --constraint 'node.labels.tenant==acme-corp' \ acme-app:latest docker service create \ --constraint 'node.labels.tenant==globex' \ globex-app:latest
This physically separates tenant workloads.
Complex Scheduling Scenarios
Geographic Distribution with Local Preference
docker service create \ --replicas 9 \ --constraint 'node.labels.region==us-east' \ --placement-pref 'spread=node.labels.zone' \ regional-service:latest
All replicas stay in us-east region (constraint) but spread across availability zones (preference).
Performance Tier Matching
# High-performance nodes docker node update --label-add performance=high node1 docker node update --label-add performance=high node2 # Medium-performance nodes docker node update --label-add performance=medium node3 docker node update --label-add performance=medium node4 # Critical service on high-performance nodes docker service create \ --constraint 'node.labels.performance==high' \ critical-api:latest # Standard service on medium-performance nodes docker service create \ --constraint 'node.labels.performance==medium' \ standard-api:latest
Storage Type Matching
# Database requiring local NVMe docker service create \ --constraint 'node.labels.storage==nvme' \ --constraint 'node.labels.storage-mount==/mnt/nvme' \ high-iops-db:latest # Archive service on spinning disks docker service create \ --constraint 'node.labels.storage==hdd' \ archive-service:latest
Handling Scheduling Failures
When no nodes satisfy constraints, tasks remain pending:
docker service ps myservice
Shows tasks in "Pending" state with a message like "no suitable node (insufficient resources on 3 nodes)".
Troubleshooting Pending Tasks
Check node labels:
docker node inspect worker-01 --format '{{.Spec.Labels}}'
Verify node availability:
docker node ls
Review service constraints:
docker service inspect myservice --format '{{.Spec.TaskTemplate.Placement}}'
Common issues:
- Typos in label names or values
- No nodes have required labels
- All matching nodes at capacity
- All matching nodes are paused or draining
Dynamic Label Management
Labels can be updated while services run:
# Add new label to node docker node update --label-add newlabel=value worker-01
Existing tasks don't move automatically. To apply new label-based placement:
# Force service update to trigger rescheduling docker service update --force myservice
This recreates tasks, respecting current constraints and preferences.
Removing Labels
# Remove label from node docker node update --label-rm oldlabel worker-01
If running tasks relied on this label via constraints, they continue running but new replicas can't be placed on this node.
Platform-Specific Scheduling
Schedule based on operating system or architecture:
# Linux-only service docker service create \ --constraint 'node.platform.os==linux' \ linux-app:latest # ARM architecture docker service create \ --constraint 'node.platform.arch==arm64' \ arm-optimized:latest # Windows nodes docker service create \ --constraint 'node.platform.os==windows' \ windows-app:latest
Useful in heterogeneous clusters with mixed node types.
Scheduler Efficiency Considerations
The scheduler evaluates all active nodes for each task placement. In large clusters with complex constraints:
- Keep constraint expressions simple
- Use labels judiciously—too many labels create management overhead
- Spread preferences are more expensive than simple constraints
- The scheduler caches node state but re-evaluates on each scheduling decision
For optimal performance, design label hierarchies that minimize the number of constraints per service.
Best Practices for Constraints
Use descriptive label names:
# Good docker node update --label-add datacenter=us-east-1 docker node update --label-add storage-type=ssd # Avoid docker node update --label-add dc=use1 docker node update --label-add st=s
Establish label standards: Document your label schema and enforce consistent naming across the cluster.
Version labels: Include version information when labels represent capabilities:
docker node update --label-add cuda-version=11.2 docker node update --label-add kernel-version=5.10
Use hierarchical labels: Create label hierarchies for complex infrastructure:
docker node update --label-add region=us-east docker node update --label-add zone=us-east-1a docker node update --label-add rack=rack-42
Avoid over-constraining: Too many constraints make scheduling brittle. Start with broader constraints and add specificity only when needed.
Test constraints: Before deploying critical services, verify constraints work as expected:
docker service create --replicas 1 --constraint 'node.labels.test==value' alpine sleep 3600 docker service ps test-service
Best Practices for Preferences
Prefer spread over constraints: When distribution is desired but not mandatory, use preferences rather than constraints.
Layer preferences: Use multiple spread preferences to create sophisticated distribution patterns:
--placement-pref 'spread=node.labels.datacenter' \ --placement-pref 'spread=node.labels.rack' \ --placement-pref 'spread=node.labels.host'
Balance with replica count: Ensure sufficient replicas to benefit from spread preferences. Spreading 3 replicas across 10 datacenters provides limited benefit.
Monitor spread effectiveness: Periodically verify that spread preferences achieve desired distribution:
docker service ps myservice --format "{{.Node}}"
Docker Swarm's constraint and preference system provides powerful, flexible control over task placement. Constraints enforce mandatory requirements, ensuring tasks only run on suitable nodes, while preferences guide the scheduler toward optimal placement without making distribution mandatory. By combining constraints for absolute requirements with preferences for distribution goals, and by carefully designing node label hierarchies, you can create sophisticated scheduling strategies that optimize workload placement for performance, fault tolerance, and resource utilization across your Swarm cluster.