Docker Swarm's architecture is built on distributed systems principles that enable reliable container orchestration across multiple hosts. At its core, Swarm uses the Raft consensus algorithm to maintain consistent cluster state across manager nodes, while worker nodes execute the actual container workloads. Understanding this internal architecture—how managers coordinate through Raft, how the cluster maintains state, and how workers receive and execute tasks—is essential for operating production Swarm clusters effectively.

The Two-Tier Node Architecture

Docker Swarm clusters consist of two distinct node types, each serving specific functions within the orchestration system:

Manager nodes handle cluster orchestration, maintain cluster state, and make scheduling decisions. They run the Raft consensus protocol to ensure consistency across the management plane.

Worker nodes execute container tasks assigned by managers. They report status back to managers but don't participate in cluster management decisions.

This separation of concerns allows Swarm to scale the control plane independently from the data plane, supporting large clusters with many workers managed by a smaller set of managers.

Understanding the Raft Consensus Algorithm

Raft is a consensus algorithm that enables distributed systems to agree on shared state even when some nodes fail. Swarm uses Raft to maintain consistent cluster state across all manager nodes.

Why Consensus Matters

In distributed systems, multiple nodes must agree on the current state of the cluster. Without consensus, different managers might have conflicting views of which services should run where, leading to inconsistent behavior.

Raft solves this by ensuring all managers agree on every state change before applying it. If managers disagree or communication fails, Raft's protocol resolves conflicts deterministically.

The Leader-Follower Model

Raft organizes manager nodes into a leader-follower hierarchy:

Leader: Exactly one manager acts as the Raft leader at any time. The leader handles all client requests, makes scheduling decisions, and coordinates state changes.

Followers: All other managers are followers. They replicate the leader's state and can become leader if the current leader fails.

Candidates: During leader election, followers temporarily become candidates until a new leader is chosen.

This model ensures there's always a single source of truth for cluster state while maintaining fault tolerance through redundancy.

Leader Election Process

When a Swarm cluster initializes or when the current leader fails, managers execute a leader election:

Timeout triggers election: Followers wait for heartbeats from the leader. If a follower doesn't receive heartbeats within the election timeout, it becomes a candidate.
Candidate requests votes: The candidate increments its term number and requests votes from other managers.
Managers cast votes: Each manager can vote for one candidate per term. Managers vote for the first candidate that requests their vote.
Majority wins: A candidate becomes leader when it receives votes from a majority of managers (more than half).
Leader announces: The new leader sends heartbeats to all followers, establishing its authority.

Election timeouts are randomized (typically 150-300ms) to reduce the probability of split votes where multiple candidates compete simultaneously.

The Raft Log

Raft maintains an append-only log of all state changes. Every operation that modifies cluster state—creating services, updating tasks, modifying node labels—is recorded as a log entry.

Log Structure

Each log entry contains:

Term number: The leader's term when the entry was created
Index: The entry's position in the log
Command: The actual state change being recorded
Timestamp: When the entry was created

The log grows monotonically. Entries are never deleted or modified, only appended.

Log Replication

When the leader receives a request to change cluster state:

Leader creates log entry: The change is recorded as a new log entry with the current term and next available index.
Leader replicates to followers: The leader sends the new entry to all follower managers.
Followers acknowledge: Each follower appends the entry to its log and sends acknowledgment to the leader.
Leader commits: Once a majority of managers have replicated the entry, the leader marks it as committed.
State application: The leader applies the committed entry to its state machine and notifies followers to do the same.
Response to client: Only after committing the entry does the leader respond to the original request.

This process ensures that state changes are durable and consistent across managers before being applied.

Term Numbers and Log Consistency

Raft uses term numbers to maintain log consistency:

Term: A logical time period during which one leader serves. Each time a new leader is elected, the term number increments.

Term comparison: When managers communicate, they compare term numbers. A manager with a lower term knows it's outdated and updates itself.

Log matching: Two logs with entries at the same index and term are guaranteed to contain identical entries up to that point.

These properties ensure that even if leaders change, the log remains consistent across all managers.

Committed vs Uncommitted Entries

Log entries exist in two states:

Uncommitted: The entry is in the leader's log but hasn't been replicated to a majority of managers. If the leader fails, this entry might be lost.

Committed: The entry has been replicated to a majority of managers. It's now durable and will never be lost, even if leaders fail.

Only committed entries are applied to the cluster's state. This ensures that state changes persist even through leader failures.

The State Machine

Each manager node runs a state machine that represents the current cluster state. This includes:

All services and their configurations
All tasks and their assignments
All nodes and their attributes
All network and volume definitions

The state machine is deterministic: given the same sequence of log entries, all managers produce identical state. This is what makes Raft work—managers replay the same log to arrive at the same state.

How Workers Fit Into the Architecture

Worker nodes don't participate in Raft consensus. Instead, they:

Connect to managers: Workers maintain connections to manager nodes.
Report capabilities: Workers inform managers about their resources (CPU, memory, disk) and attributes.
Receive task assignments: Managers tell workers which containers to run.
Execute tasks: Workers start, stop, and monitor containers as directed.
Report status: Workers continuously update managers on task status.

Workers are stateless from the orchestration perspective—all cluster state lives in the Raft log on managers.

Manager Node Responsibilities

Manager nodes handle numerous orchestration functions:

Task Scheduling

Managers decide which workers should run which tasks based on:

Available resources
Service requirements
Current cluster state

The leader manager makes all scheduling decisions, ensuring consistency.

Service Management

Managers maintain the desired state for all services:

Tracking which tasks should be running
Detecting when tasks fail
Initiating task replacements
Handling service updates

Cluster State Management

Managers store and maintain:

Service definitions
Task states and history
Node information
Network configurations
Volume definitions

All this state is replicated through Raft for durability.

Manager to Worker Communication

Managers and workers communicate through several channels:

Task assignment: Managers send task specifications to workers, describing which containers to run with what configurations.

Status reporting: Workers continuously stream status updates back to managers, reporting task states and health.

Resource updates: Workers report their available resources, allowing managers to make informed scheduling decisions.

Heartbeats: Workers send regular heartbeats proving they're alive and functioning.

If a worker stops sending heartbeats, managers detect the failure and reschedule its tasks elsewhere.

Quorum Requirements

Raft requires a quorum (majority) of managers to be available for the cluster to function:

3 managers: Tolerates 1 failure (quorum: 2) 5 managers: Tolerates 2 failures (quorum: 3) 7 managers: Tolerates 3 failures (quorum: 4)

The formula is: quorum = (n/2) + 1 where n is the total number of managers.

If quorum is lost:

Existing tasks continue running on workers
No new tasks can be scheduled
No cluster state changes can be made
The cluster is effectively read-only

Quorum ensures that state changes are durable and that split-brain scenarios (multiple conflicting leaders) cannot occur.

The Raft Snapshot Mechanism

As the Raft log grows, replaying it from the beginning becomes expensive. Raft uses snapshots to compact the log:

Snapshot creation: Periodically, managers create a snapshot of the current state machine.

Log truncation: Once a snapshot is created, log entries before the snapshot point can be discarded.

Snapshot replication: New managers joining the cluster receive the latest snapshot plus any subsequent log entries, rather than replaying the entire log history.

Snapshots reduce the time needed to restore manager state after failures or when adding new managers.

Manager Node Internals

Each manager node runs several internal components:

The Raft Engine

Implements the Raft protocol, handling:

Leader election
Log replication
Commitment decisions

The Orchestrator

Makes scheduling and placement decisions for tasks. The orchestrator runs only on the leader manager.

The Dispatcher

Communicates with worker nodes, sending task assignments and receiving status updates.

The Allocator

Manages IP address allocation for services and tasks within overlay networks.

The Store

Maintains the current cluster state, including services, tasks, nodes, and configurations.

Worker Node Internals

Worker nodes are simpler than managers:

The Agent

Connects to managers and handles bidirectional communication, receiving task assignments and sending status updates.

The Executor

Starts and stops containers based on task specifications received from managers. The executor interacts directly with the Docker Engine.

The Reporter

Continuously monitors task status and reports state changes back to managers.

Workers don't make decisions—they faithfully execute instructions from managers.

How a Service Starts: The Complete Flow

Understanding the complete lifecycle of service creation illustrates how all components work together:

Client request: An administrator executes docker service create, sending the request to any manager node.
Leader processing: If the request hits a follower, it forwards the request to the leader. The leader processes all write requests.
Log entry creation: The leader creates a log entry representing the new service and replicates it to followers.
Commitment: Once a majority of managers have the log entry, the leader commits it.
State application: All managers apply the committed entry, updating their state machines with the new service definition.
Task creation: The orchestrator (running on the leader) creates task objects for the service's replicas.
Task scheduling: The orchestrator assigns tasks to specific worker nodes based on availability and requirements.
Task dispatch: The dispatcher sends task assignments to the selected workers.
Container creation: Workers receive their task assignments and start the specified containers.
Status reporting: Workers report task status back to managers, updating the cluster state.
Continuous reconciliation: Managers continuously compare desired state (from the Raft log) with actual state (from worker reports) and take corrective action when they diverge.

Manager Communication Patterns

Managers communicate through specific patterns:

Leader to followers: The leader sends log entries and heartbeats to all followers. This is unidirectional—followers don't send data back during normal operation.

Followers to leader: Followers send acknowledgments for log entries and vote requests during elections.

Manager discovery: Managers discover each other through a gossip protocol, maintaining a view of all managers in the cluster.

Client requests: Followers forward write requests to the leader but can serve read-only requests themselves.

The Role of etcd in Swarm

Docker Swarm's Raft implementation is custom-built for Swarm's specific needs. Unlike some orchestrators that use external etcd clusters, Swarm embeds its Raft implementation directly into manager nodes.

This design simplifies operations—there's no separate distributed key-value store to manage. All cluster state lives in the Raft log, replicated across managers.

Initializing a Swarm Cluster

When you initialize a Swarm cluster with docker swarm init:

First manager: The initializing node becomes the first manager and the initial Raft leader.
Bootstrap term: The cluster starts at term 1.
Empty log: The Raft log begins empty, with only bootstrap configuration.
Join tokens: The manager generates cryptographic tokens used by other nodes to join the cluster.
State initialization: The state machine is initialized with the first manager's information.

The first manager immediately achieves quorum (majority of 1 is 1) and becomes operational.

Adding Managers to a Cluster

When a new manager joins:

Join request: The new node connects using the manager join token.
Authentication: The join token is verified cryptographically.
State transfer: The new manager receives the current Raft snapshot and any subsequent log entries.
Log replay: The new manager replays the log to build its state machine.
Follower status: The new manager becomes a follower, participating in log replication.
Quorum update: The cluster's quorum requirement increases with the new manager.

The cluster continues operating during this process without interruption.

Manager Failure and Recovery

When a manager fails:

Non-leader failure: If a follower fails, the cluster continues operating normally as long as quorum remains. The leader continues replicating logs to remaining followers.

Leader failure: If the leader fails, followers detect the missing heartbeats and initiate a new election. Once a new leader is elected, the cluster resumes normal operation.

Temporary failure: If a manager temporarily loses network connectivity, it falls behind in log replication. When reconnected, it catches up by receiving missing log entries from the leader.

Permanent failure: If a manager is permanently lost, it should be removed from the cluster to adjust quorum requirements.

Worker Node Registration

When workers join the cluster:

Join request: The worker connects using the worker join token.
Authentication: The token is verified by any manager.
Registration: The manager creates a node object in the cluster state, recording the worker's capabilities.
Raft replication: The new node information is replicated through Raft to all managers.
Agent connection: The worker establishes persistent connections to multiple managers for redundancy.
Resource advertisement: The worker reports its available CPU, memory, and other resources.
Ready state: Once registered, the worker is available for task scheduling.

Workers can connect to any manager, and the system ensures their registration is consistently recorded across all managers.

Data Persistence

Swarm persists its Raft log and state to disk on manager nodes at /var/lib/docker/swarm. This directory contains:

Raft log: The complete history of cluster state changes Snapshots: Periodic snapshots of cluster state Certificates: Cryptographic material for manager authentication WAL: Write-ahead log for durability

If a manager restarts, it reads this data to restore its state without needing to rejoin the cluster.

The Importance of Disk Durability

Raft's consistency guarantees depend on durable storage. When a manager acknowledges a log entry, it must have been written to disk. If managers use non-durable storage (like tmpfs), log entries could be lost, potentially leading to inconsistency.

For production clusters, managers should use persistent storage with proper fsync support to ensure Raft's durability guarantees.

Manager Resource Requirements

Managers have modest resource requirements because they handle orchestration, not workload execution:

CPU: 1-2 cores sufficient for typical clusters Memory: 1-2GB for the Raft log and state machine Disk: Depends on cluster size; typically a few GB for the Raft log Network: Managers need reliable, low-latency connectivity to each other

The Raft log grows over time, so monitors should track disk usage and implement log rotation or cleanup strategies.

Cluster Topology Considerations

Manager placement affects cluster resilience:

Single datacenter: Place managers on different physical hosts to survive individual host failures.

Multiple datacenters: Distribute managers across locations to survive datacenter failures, but ensure low-latency connectivity between managers for Raft performance.

Network partitions: If managers are split by a network partition such that neither side has quorum, the cluster stops accepting changes until connectivity is restored.

Understanding these failure modes helps design resilient cluster topologies.

Docker Swarm's architecture elegantly separates concerns: Raft provides consistent, fault-tolerant state management across managers, while workers focus solely on executing containers. This design enables reliable, distributed container orchestration with clear failure boundaries and understandable operational characteristics. The Raft consensus algorithm ensures that even when managers fail or network problems occur, the cluster maintains a consistent view of desired state and continues operating as long as quorum is maintained.

Docker Swarm Internal Architecture: Raft, Managers, and Workers