Rolling updates enable you to update running services with zero downtime, gradually replacing old versions with new ones while maintaining service availability. This capability is essential for continuous deployment and operational flexibility in production environments.
Understanding Rolling Updates
A rolling update is a gradual replacement strategy that updates a service incrementally rather than all at once. Instead of stopping all tasks and starting new ones simultaneously, rolling updates stop and replace tasks in batches, ensuring some tasks remain running throughout the update process.
Why Rolling Updates Matter
Without rolling updates, updating a service means complete downtime: stop all tasks, start new ones, and wait for them to become healthy. During this window, your service is unavailable.
Rolling updates eliminate this downtime. By updating incrementally, some tasks remain available to serve traffic while others are being updated. Users experience no interruption, even during active updates.
The Update Process
During a rolling update, the orchestrator stops a subset of tasks, starts replacement tasks with the new configuration, waits for them to become healthy, then proceeds to the next batch. This continues until all tasks are updated.
The orchestrator manages this process automatically. You specify what should change and how the update should proceed, and the orchestrator handles the detailed execution.
Triggering Rolling Updates
Updates are triggered when you change a service's configuration:
Image Updates
The most common update is changing the service image to deploy new application versions:
docker service update --image myapp:v2.0 myapp
This command initiates an update, replacing the current image with the new version.
Configuration Updates
Other changes also trigger updates: environment variables, command arguments, resource limits, or mount configurations:
docker service update --env-add NEW_VAR=value myapp docker service update --limit-cpu 0.5 myapp
Any change that requires recreating containers triggers a rolling update.
Update Configuration Parameters
Several parameters control how updates proceed:
Parallelism
Parallelism determines how many tasks are updated simultaneously:
docker service update --update-parallelism 2 myapp
With parallelism of 2, the orchestrator updates 2 tasks at a time. Higher parallelism means faster updates but more tasks unavailable simultaneously. Lower parallelism means slower updates but more tasks remain available.
Update Delay
The delay specifies how long to wait between batches:
docker service update --update-delay 10s myapp
After updating one batch, the orchestrator waits 10 seconds before starting the next batch. This delay allows new tasks to stabilize and allows monitoring for problems before proceeding.
Delays can be specified in seconds (s), minutes (m), or hours (h).
Update Order
The --update-order parameter controls whether new tasks are started before or after old tasks are stopped:
docker service update --update-order start-first myapp
- stop-first (default): Stop old tasks, then start new tasks. This never exceeds desired replica count but may temporarily reduce available capacity.
- start-first: Start new tasks, then stop old tasks. This maintains capacity but temporarily exceeds the desired replica count, requiring extra resources.
Update Monitoring
The orchestrator monitors updates to detect failures:
Monitor Duration
The --update-monitor parameter specifies how long to monitor new tasks before considering them successfully updated:
docker service update --update-monitor 30s myapp
After starting a new task, the orchestrator waits 30 seconds. If the task remains healthy during this period, it's considered successfully updated. If it fails, the update is considered failed.
Failure Detection
The orchestrator detects failures through multiple mechanisms:
- Tasks that exit immediately after starting
- Tasks that fail health checks
- Tasks that don't reach running state within the monitor period
Any of these conditions indicates an update failure for that task.
Handling Update Failures
When updates fail, the orchestrator takes action based on configured failure behavior:
Failure Actions
The --update-failure-action parameter controls what happens on failure:
docker service update --update-failure-action rollback myapp
- pause: Stop the update, leaving some tasks on old version and some on new. Requires manual intervention.
- continue: Continue updating despite failures. Useful when some task failures are acceptable.
- rollback: Automatically revert to the previous configuration, undoing the update.
Maximum Failure Ratio
The --update-max-failure-ratio parameter defines what constitutes update failure:
docker service update --update-max-failure-ratio 0.3 myapp
If more than 30% of updated tasks fail, the entire update is considered failed, and the failure action is triggered.
This prevents rolling out broken updates. If individual task failures are rare, they won't stop the update. If many tasks fail, indicating a systemic problem, the update stops.
Rollback Capability
The orchestrator maintains the previous service configuration, enabling rollbacks:
Automatic Rollback
When --update-failure-action rollback is configured, the orchestrator automatically reverts to the previous configuration if the update fails.
The rollback uses the same batching parameters as the update: parallelism, delay, and order.
Manual Rollback
You can manually trigger a rollback at any time:
docker service rollback myapp
This reverts the service to its previous configuration, using a rolling process to minimize disruption.
Rollback Configuration
Rollback behavior can be configured independently from update behavior:
docker service update \ --rollback-parallelism 3 \ --rollback-delay 5s \ --rollback-monitor 20s \ myapp
These parameters work like their update equivalents but control rollback behavior specifically.
Update Strategies
Different update strategies suit different scenarios:
Conservative Updates
Use low parallelism (1 or 2) and longer delays (30s-60s):
docker service update \ --update-parallelism 1 \ --update-delay 60s \ --update-monitor 30s \ --image myapp:v2.0 \ myapp
This maximizes safety, giving you time to detect and respond to problems before many tasks are updated.
Aggressive Updates
Use high parallelism and short delays:
docker service update \ --update-parallelism 5 \ --update-delay 5s \ --update-monitor 10s \ --image myapp:v2.0 \ myapp
This completes updates quickly but risks updating many tasks before problems are detected.
Balanced Updates
Use moderate parallelism (2-3) and reasonable delays (10-30s):
docker service update \ --update-parallelism 2 \ --update-delay 20s \ --update-monitor 30s \ --image myapp:v2.0 \ myapp
This balances update speed with safety, suitable for most production environments.
Update Performance Impact
Updates affect service performance in predictable ways:
Capacity Reduction
With stop-first ordering, capacity temporarily decreases during updates. If parallelism is 2 out of 10 total tasks, capacity is reduced by 20% while those 2 tasks are being replaced.
Higher parallelism means larger capacity reductions. Ensure your service can handle the temporary reduction without performance degradation.
Capacity Increase
With start-first ordering, capacity temporarily increases. New tasks start before old tasks stop, meaning you momentarily have more tasks than desired.
This requires extra resources. Ensure hosts have capacity for the temporary additional tasks.
Connection Disruption
When tasks stop, their existing connections are closed. Clients must reconnect, potentially causing brief errors or retries.
Graceful shutdown procedures in applications minimize this disruption, allowing in-flight requests to complete before tasks exit.
Update Timing Considerations
Off-Peak Updates
Schedule updates during low-traffic periods when temporary capacity reduction has minimal impact. This provides a safety margin if problems occur.
Business Hours Updates
Conversely, updating during business hours ensures staff are available to respond if problems arise. This trades off user impact risk for response capability.
Staged Updates
For critical services, consider staging updates: update a subset of tasks first, monitor for issues, then update the remainder. This manual approach provides maximum control.
Monitoring During Updates
Active monitoring during updates enables quick problem detection:
Task Health
Monitor task health throughout the update. Watch for tasks failing health checks or exiting unexpectedly. These indicate the new version has problems.
Error Rates
Track application error rates during updates. If error rates spike, the new version may have bugs not caught in testing.
Performance Metrics
Monitor performance metrics: response times, throughput, resource utilization. Degradation indicates the new version has performance problems.
Rollback Readiness
Be prepared to rollback quickly if problems are detected. Have rollback commands ready and ensure you can execute them immediately.
Update Best Practices
Test Before Updating
Thoroughly test new versions before updating production services. Staging environments should closely mirror production to catch issues before they affect users.
Use Specific Image Tags
Never use :latest tags for production services. Use specific version tags (:v2.0.1) so you know exactly what version is deployed and can reliably rollback if needed.
Configure Appropriate Monitoring
Set monitor duration long enough to detect issues. If your application takes 20 seconds to fully start, a 10-second monitor might not catch startup failures.
Set Conservative Failure Ratios
Use low failure ratios (0.1-0.3) to catch problems early. High ratios allow too many bad tasks before triggering failure actions.
Enable Automatic Rollback
Configure automatic rollback for production services. This provides automatic recovery if updates fail, minimizing user impact.
Document Rollback Procedures
Even with automatic rollback, document manual rollback procedures. Automatic rollback might fail or be insufficient for some situations.
Update Workflow
A typical update workflow involves preparation, execution, monitoring, and verification:
Pre-Update Preparation
Before updating, verify the new image is available and correct. Check current service health to ensure you're starting from a healthy baseline.
Review update configuration: parallelism, delays, failure actions. Ensure they're appropriate for the current situation.
Executing the Update
Trigger the update with appropriate parameters:
docker service update \ --image myapp:v2.0.1 \ --update-parallelism 2 \ --update-delay 20s \ --update-monitor 30s \ --update-failure-action rollback \ --update-max-failure-ratio 0.2 \ myapp
The orchestrator begins the rolling update immediately.
Active Monitoring
Watch the update progress closely. Monitor task states, error rates, and performance metrics. Be ready to manually rollback if automatic mechanisms don't trigger despite evident problems.
Post-Update Verification
After the update completes, verify all tasks are healthy and running the new version. Check that error rates and performance metrics have returned to normal levels.
Monitor for several hours after updates. Some issues only appear under sustained load or specific usage patterns.
Update Complexity
Update complexity varies by service type:
Stateless Services
Stateless services are simplest to update. Tasks are interchangeable, and there's no concern about data consistency or state migration. Rolling updates work seamlessly.
Stateful Services
Stateful services require careful update planning. Ensure new versions can read data written by old versions. Consider whether tasks need to coordinate during updates.
For databases or other data services, updates might require additional steps beyond simple rolling updates: data migration, schema updates, or coordinated cutover.
Services with Dependencies
When updating services with dependencies, update order matters. Update backend services before frontends that depend on them, ensuring compatibility throughout.
Update Parallelism Trade-offs
Choosing parallelism involves balancing competing concerns:
High Parallelism
Advantages: Faster updates, less time in mixed-version state.
Disadvantages: More capacity reduction, less time to detect problems, higher risk if new version is broken.
Low Parallelism
Advantages: Minimal capacity reduction, more time to detect problems, easier to stop if issues arise.
Disadvantages: Slower updates, longer time in mixed-version state.
Optimal Parallelism
Optimal parallelism depends on:
- Service criticality: More critical services use lower parallelism
- Task count: More tasks can use higher parallelism while maintaining capacity
- Confidence in new version: Well-tested versions can use higher parallelism
- Monitoring capability: Better monitoring enables higher parallelism with maintained safety
Update Delays
Update delays provide breathing room between batches:
Purpose of Delays
Delays allow new tasks to reach steady state before more tasks are updated. Tasks might start successfully but fail under load—delays increase the chance of detecting such issues.
Delays also allow monitoring systems to detect anomalies. Metrics often take time to aggregate and alert.
Choosing Delay Duration
Short delays (5-10s) provide minimal buffering. Suitable when confidence is high and monitoring is excellent.
Medium delays (20-40s) balance speed with safety. Suitable for most production updates.
Long delays (60s+) maximize safety at the cost of update speed. Suitable for critical services or risky updates.
Failure Ratio Interpretation
The failure ratio threshold determines update sensitivity:
Low Thresholds (0.1-0.2)
Low thresholds are sensitive—they stop updates if few tasks fail. This maximizes safety but might abort updates due to transient issues.
Use low thresholds for critical services or when deploying uncertain changes.
Medium Thresholds (0.3-0.4)
Medium thresholds tolerate occasional failures while still catching systemic problems. This balances safety with update success.
Suitable for most production services under normal circumstances.
High Thresholds (0.5+)
High thresholds allow many failures before stopping updates. This might be appropriate when some task failures are expected or when continuing despite issues is preferred to stopping.
Generally, high thresholds are too permissive for production use.
Update Order Implications
Update order has resource and availability implications:
Stop-First Order
Stop-first temporarily reduces capacity. Ensure your remaining tasks can handle the load. If capacity is already tight, stop-first might cause performance degradation.
Stop-first never exceeds desired replica count, simplifying resource planning.
Start-First Order
Start-first maintains capacity by starting new tasks before stopping old ones. This requires extra resources—hosts must have capacity for additional tasks.
Start-first is ideal when capacity is critical and resources are available.
Rollback Considerations
Rollbacks are safety nets, but they have limitations:
Data Compatibility
Rolling back software doesn't rollback data. If the new version wrote data in a new format, the old version might not understand it.
Design updates with backward compatibility to ensure rollbacks work correctly.
State Consistency
For stateful services, rollbacks might leave inconsistent state. Some tasks might have processed requests using new logic while others used old logic.
Careful design minimizes these issues, but they're inherent to rolling updates of stateful systems.
Rollback Timing
Automatic rollbacks occur during the update. Manual rollbacks can happen any time after the update, even hours or days later if issues emerge gradually.
Advanced Update Patterns
Canary Updates
Update a single task first (parallelism 1), monitor extensively, then update the rest if successful. This catches issues with minimal impact.
Blue-Green Updates
Maintain two full deployments (blue and green). Update green while blue serves traffic. When green is verified, switch traffic from blue to green.
This requires external load balancer integration and double resources, but provides instant rollback capability.
Phased Rollouts
Update in multiple phases with different parallelism. Start with low parallelism for initial batches, then increase parallelism as confidence grows.
Update Documentation
Document your update procedures:
Update Runbooks
Create runbooks describing update procedures for each service: pre-update checks, update commands, monitoring focus points, rollback procedures.
Update History
Maintain an update history: when updates occurred, which versions were deployed, what parameters were used, whether issues arose.
This history helps identify patterns and improve future updates.
Version Tracking
Track which versions are deployed in each environment. This prevents confusion during incidents and ensures consistency across environments.
Update Automation
While manual updates work for small deployments, automation becomes essential at scale:
Continuous Deployment
Integrate updates into CI/CD pipelines. When new versions pass tests, automatically deploy them using appropriate update configurations.
Automated Monitoring
Automate monitoring during updates. Scripts can watch metrics and trigger manual intervention or rollbacks if thresholds are exceeded.
Progressive Deployment
Implement progressive deployment: automatically update a small percentage of tasks, monitor, then automatically proceed to full update if metrics are good.
Common Update Problems
Updates Stuck in Progress
If updates hang, tasks might be failing to start or failing health checks. Inspect task logs to identify the issue.
Check resource availability—if hosts lack resources, new tasks can't be scheduled.
Frequent Task Failures
If tasks fail repeatedly during updates, the new version likely has a problem. Manually rollback and investigate the issue.
Slow Updates
If updates take too long, consider increasing parallelism or reducing delays. Ensure these changes don't compromise safety.
Inconsistent Behavior
If behavior is inconsistent during updates, you have mixed versions serving traffic. This is normal during rolling updates but can cause confusion. Ensure your application handles version mixing gracefully.
Rolling updates are essential for maintaining availability while deploying changes. Understanding update configuration, monitoring update progress, and being prepared to rollback enables you to confidently update production services with minimal risk and disruption.