Building a cluster requires adding nodes beyond the initial setup. The process of joining nodes to an existing cluster is fundamental to creating distributed infrastructure. Understanding join tokens, the join command, and the underlying mechanics ensures you can confidently expand your cluster while maintaining security and stability.
Understanding Join Tokens
Join tokens are cryptographic credentials that authorize nodes to join your cluster. When a cluster is created, two join tokens are automatically generated: one for adding nodes with management capabilities, and another for adding nodes that execute workloads. These tokens serve as authentication mechanisms, ensuring only authorized hosts can become part of your cluster.
A join token consists of three parts separated by hyphens. The prefix identifies it as a join token. The middle section contains encoded information about the token type and cluster identity. The final section is a secret value that provides the actual authentication credential. This structure allows the system to validate tokens before processing join requests.
SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h
The security of your cluster depends on protecting these tokens. Anyone with a valid join token can add a node to your cluster. If tokens are compromised, unauthorized nodes could join, potentially accessing sensitive data or consuming resources. Treat join tokens as you would passwords or API keys.
Retrieving Join Tokens
After initial cluster setup, you'll need to retrieve join tokens to add additional nodes. The system provides commands to display current tokens on demand. These commands must be executed on an existing cluster member that has management privileges.
To retrieve the token for adding execution nodes:
docker swarm join-token worker
This command displays the complete join command, including the token and the address where new nodes should connect:
To add a worker to this swarm, run the following command:
docker swarm join --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h 192.168.1.100:2377
Similarly, to retrieve the token for adding management nodes:
docker swarm join-token manager
The output format is identical, but the token itself differs. The system knows which type of token is being used based on its encoded contents. Using the wrong token for the intended node type won't work—the token explicitly defines what role the joining node will have.
The Join Command Structure
The join command executed on the new node connects it to the cluster. The basic structure requires the join token and the address of an existing cluster member:
docker swarm join --token <TOKEN> <HOST>:<PORT>
The token authenticates the join request. The host and port specify where to connect—typically an existing management node's address. Port 2377 is the standard cluster management port, though custom configurations might use different ports.
When executed, this command transforms the standalone Docker installation into a cluster member. The Docker daemon on the joining node establishes communication with the cluster, receives its configuration, and begins participating in cluster operations.
Joining as Different Node Types
The token determines what type of node is created. Using a token retrieved with docker swarm join-token worker creates an execution node. Using a token from docker swarm join-token manager creates a management node. The join command itself is identical—only the token changes.
Example of joining as an execution node:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ 192.168.1.100:2377
Example of joining as a management node:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-2a3b4c5d6e7f8g9h0i1j2k3l \ 192.168.1.100:2377
The only difference is the token value. Everything else about the command remains the same. This simplicity makes it easy to add nodes, but also means you must be careful to use the correct token for your intended node type.
Specifying Advertise Address During Join
Just as during cluster creation, nodes joining the cluster need to advertise how other nodes should reach them. The --advertise-addr flag specifies this address:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ --advertise-addr 192.168.1.101 \ 192.168.1.100:2377
This tells the cluster that other nodes should use 192.168.1.101 to communicate with this joining node. Without this flag, Docker attempts to auto-detect an appropriate address. Auto-detection often works, but explicit specification is more reliable, especially on hosts with multiple network interfaces.
The advertise address should be reachable by all other cluster members. Using localhost or 127.0.0.1 won't work because these addresses are only valid on the local machine. Other nodes trying to reach 127.0.0.1 would connect to themselves, not to the joining node.
Listen Address Configuration
The --listen-addr flag controls which address and port the node binds to for cluster communication:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ --advertise-addr 192.168.1.101 \ --listen-addr 0.0.0.0:2377 \ 192.168.1.100:2377
The default listen address is 0.0.0.0:2377, meaning the node listens on all network interfaces on port 2377. This default works for most scenarios. Specifying a different listen address is useful when you want to restrict which interface handles cluster traffic.
The listen and advertise addresses serve different purposes. Listen address determines where the node actually accepts connections. Advertise address tells other nodes where to connect. In most cases, you advertise a specific IP while listening on all interfaces (0.0.0.0).
Data Path Address for Joining Nodes
Container-to-container communication can use a different network path than cluster management traffic. The --data-path-addr flag specifies the address for data plane traffic:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ --advertise-addr 192.168.1.101 \ --data-path-addr 10.0.0.101 \ 192.168.1.100:2377
In this example, cluster management happens via 192.168.1.101, but container traffic uses 10.0.0.101. This separation allows you to use different networks for different purposes—perhaps a management network and a higher-bandwidth data network.
If you don't specify a data path address, the node uses the advertise address for both management and data traffic. For many deployments, this single-network approach is perfectly adequate.
Availability State During Join
Nodes can join with a specific availability state. The --availability flag controls whether the joining node should immediately accept workload:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ --advertise-addr 192.168.1.101 \ --availability drain \ 192.168.1.100:2377
Three availability states exist:
- active: The node accepts workload (default)
- pause: The node doesn't accept new workload but keeps existing workload running
- drain: The node doesn't run any workload
Setting availability to drain during join is useful when you're adding nodes that should only handle management duties, or when you're adding nodes that aren't yet ready for production traffic.
Join Process Mechanics
When you execute the join command, several operations occur behind the scenes. The joining node first establishes a TLS connection to the address specified in the command. It presents the join token for authentication.
The receiving node validates the token. If valid, it forwards the join request to the cluster's consensus system. The consensus system records the new node's existence and assigns it a unique node ID. This ID persists for the node's lifetime in the cluster.
The joining node receives a certificate from the cluster's certificate authority. This certificate identifies the node and encrypts its communications with other cluster members. Certificate issuance happens automatically as part of the join process.
Once the certificate is issued, the node is officially part of the cluster. It begins receiving information about cluster state, other nodes, and its assigned responsibilities. The entire join process typically completes in seconds.
Join Command Output
Successful join produces output confirming cluster membership:
This node joined a swarm as a worker.
Or for management nodes:
This node joined a swarm as a manager.
This simple confirmation indicates the join succeeded and identifies the node's role. The node is now fully integrated into the cluster and ready to participate in cluster operations.
Multiple Manager Addition
When adding multiple management nodes, do so sequentially rather than simultaneously. Add one management node, wait for it to fully integrate, then add the next. This serial approach ensures cluster stability during expansion.
Adding many management nodes at once can cause consensus problems. The existing managers must agree on each new addition. If multiple joins happen simultaneously, the rapid state changes can overwhelm the consensus system, causing delays or failures.
Best practice suggests always maintaining an odd number of management nodes. If you have three managers and want to add two more, add them one at a time. After adding the fourth manager (temporarily giving you four managers), immediately add the fifth to restore an odd count.
Connecting to Different Cluster Members
The join command needs at least one existing cluster member's address, but you can specify multiple addresses for redundancy:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ 192.168.1.100:2377,192.168.1.101:2377,192.168.1.102:2377
The joining node tries each address in order until one succeeds. This redundancy prevents join failure if a single cluster member is temporarily unavailable. However, at least one specified address must be reachable for the join to succeed.
While you can specify any cluster member's address, specifying management node addresses is most reliable. Management nodes are always present and can always process join requests. Execution node addresses might work, but they forward the request to a manager, adding an extra network hop.
Network and Firewall Requirements
For a node to successfully join, specific network ports must be accessible. Port 2377 (TCP) must be open for cluster management traffic. The joining node needs to reach this port on at least one existing cluster member, and other members need to reach this port on the joining node.
Port 7946 (TCP and UDP) must be open for inter-node communication. This port handles cluster state synchronization and discovery. Both the joining node and existing members need bidirectional access on this port.
Port 4789 (UDP) must be open for container network traffic. This port carries the VXLAN traffic that enables containers on different nodes to communicate. Like port 7946, it requires bidirectional access.
Firewall rules blocking these ports are a common cause of join failures. The join command might connect successfully (port 2377 is reachable), but the node fails to fully integrate because ports 7946 or 4789 are blocked. Always verify all required ports are open before attempting to join.
Join Token Rotation
Security best practices include periodically rotating join tokens. The rotation command invalidates the old token and generates a new one:
docker swarm join-token --rotate worker
This generates a new token for execution nodes. Any nodes trying to join with the old token will be rejected. Existing nodes are unaffected—rotation only impacts future join attempts.
Similarly, to rotate the management token:
docker swarm join-token --rotate manager
Token rotation is essential if a token is compromised or accidentally exposed. Perhaps it was committed to a git repository, shared in an insecure chat, or logged somewhere visible. Rotating the token immediately prevents unauthorized joins.
After rotation, you must retrieve the new token and distribute it to anyone authorized to add nodes. The old token becomes useless immediately upon rotation.
Quiet Mode for Join Tokens
By default, the join-token command displays the complete join command including explanatory text. The --quiet flag outputs only the token itself:
docker swarm join-token --quiet worker
Output:
SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h
This format is useful in automation scripts where you need just the token value. You can capture it in a variable and construct the join command programmatically:
TOKEN=$(docker swarm join-token --quiet worker) ssh user@newnode "docker swarm join --token $TOKEN 192.168.1.100:2377"
The quiet flag works identically for both execution and management tokens.
Automated Join Procedures
In cloud environments or automated infrastructure, you often want to join nodes automatically without manual intervention. This requires securely distributing the join token and cluster address to new nodes.
One approach stores the join command in a secure parameter store or secrets manager. When a new instance launches, it retrieves the join command and executes it. This keeps the token secure while enabling automation.
Another approach uses configuration management tools that distribute the join command to new nodes. The configuration management system itself authenticates to the cluster to retrieve tokens, then provides the join command to managed nodes.
For cloud providers with auto-scaling, join commands can be included in instance launch templates or user data scripts. New instances automatically join the cluster when they start, enabling the cluster to grow and shrink automatically based on load.
Handling Join Failures
Several issues can cause join failures. Understanding common problems helps troubleshoot when joins don't succeed.
Network connectivity is the most common issue. If the joining node can't reach any specified cluster address, the join fails immediately. Verify network connectivity using ping or telnet before attempting to join.
Firewall rules blocking required ports cause join failures that might not be immediately obvious. The initial connection on port 2377 might succeed, but the node fails to integrate because ports 7946 or 4789 are blocked. Check all three required ports.
Invalid or expired tokens cause authentication failures. If a token has been rotated, old tokens no longer work. Retrieve a current token and try again.
Time synchronization matters for certificate validation. If the joining node's clock is significantly out of sync with the cluster, certificate validation fails. Ensure all nodes use NTP or another time synchronization mechanism.
Docker version incompatibility can cause joins to fail. While minor version differences are usually tolerated, major version mismatches might cause problems. Verify all nodes run compatible Docker versions.
Join Command Verification
After executing a join command, verify the node successfully joined before proceeding. Check the local Docker info:
docker info
Look for the cluster status section:
Swarm: active
NodeID: abc123def456
Is Manager: false
Node Address: 192.168.1.101
Manager Addresses:
192.168.1.100:2377
This confirms the node is part of the cluster and shows its role and connection information. If this section shows "Swarm: inactive" or is missing, the join didn't succeed.
Re-joining After Leaving
If a node leaves the cluster (whether gracefully or due to failure), it can rejoin. However, rejoining creates a new node identity. The old node ID is not reused. From the cluster's perspective, this is a different node, even though it's the same physical or virtual machine.
To rejoin, simply execute the join command again:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ 192.168.1.100:2377
The node receives a new node ID and a new certificate. Any state associated with the old node ID is not automatically transferred to the new identity.
Label Assignment During Join
While labels aren't specified during the join command itself, understanding that labels will be assigned after joining is important for planning. Labels are metadata attached to nodes that enable sophisticated workload placement.
Immediately after joining, a node has minimal labels—typically just the hostname and operating system. Additional labels must be added after the node joins, using cluster management commands executed on a management node.
Planning your labeling strategy before adding nodes helps ensure consistent node identification. Decide on labels like datacenter, environment, hardware type, or purpose before joining nodes, even though you'll apply these labels after the join completes.
Security Considerations for Node Addition
Every node added to your cluster increases your security surface area. Each node is a potential entry point if compromised. Carefully control who can retrieve join tokens and who has access to systems that can join your cluster.
Join tokens should be transmitted securely. Don't send them via unencrypted email or store them in plaintext in shared documents. Use secure channels like encrypted chat, secrets managers, or secure file transfer.
Audit join events. Monitoring which nodes join your cluster and when helps detect unauthorized additions. Most production environments log cluster membership changes for security auditing.
Consider network segmentation. Nodes joining from untrusted networks pose risks. If possible, only allow joins from trusted network segments. Use VPNs or private networks for cluster communication.
Time Synchronization and Certificates
Accurate time synchronization across all nodes is critical. Certificates issued during the join process have validity periods. If a node's clock is significantly wrong, it might generate or receive certificates that appear expired or not yet valid.
Time skew of more than a few minutes can cause certificate validation failures. Use NTP (Network Time Protocol) to keep all cluster members synchronized. This is especially important in cloud environments where virtual machines might experience time drift.
Check time synchronization before joining nodes:
timedatectl status
Ensure the "System clock synchronized" shows yes and that the time displayed is accurate. If not, configure and enable NTP before attempting to join.
DNS Considerations
While join commands typically use IP addresses, DNS names are also supported:
docker swarm join \ --token SWMTKN-1-3x7p2h1q8f9k2n6m5t4w8v7u3y2x1z0-9s8r7q6p5o4n3m2l1k0j9i8h \ manager1.example.com:2377
Using DNS names can simplify cluster management, especially in cloud environments where IP addresses might change. However, DNS resolution must work reliably. If DNS fails, joins fail.
For advertise addresses, IP addresses are generally more reliable than DNS names. Other nodes need to reach the advertised address consistently, and DNS resolution adds a dependency that could fail.
Cloud Provider Considerations
Cloud platforms often have specific networking configurations that affect node joining. Virtual Private Clouds (VPCs), security groups, network ACLs, and other cloud networking features must be configured to allow cluster traffic.
In AWS, for example, security groups must allow traffic on ports 2377, 7946, and 4789 between cluster members. Nodes in different subnets must be able to reach each other on these ports.
In Azure, Network Security Groups (NSGs) serve a similar function. Ensure NSG rules permit cluster traffic.
In GCP, firewall rules must allow the required ports. Consider creating specific firewall rules for cluster traffic, using network tags or service accounts to identify cluster members.
Cloud metadata services can provide information useful during automated joins. Instance IP addresses, region information, and custom metadata can all help construct appropriate join commands automatically.
Container Runtime and Version Compatibility
The joining node's Docker installation must be compatible with the cluster's Docker version. While minor version differences are generally acceptable, large version gaps might cause problems.
Before joining a node, verify its Docker version:
docker --version
Compare this with the Docker version running on existing cluster members. If there's a significant difference, consider upgrading or downgrading the joining node to match the cluster.
Generally, it's safest to run the same Docker version across all cluster members. This eliminates version compatibility concerns and ensures all features work consistently.
Post-Join Verification
After a node joins, perform thorough verification before considering it production-ready. Check that the node appears in cluster inventories (when viewed from management nodes). Verify network connectivity to other nodes. Confirm that required ports are accessible.
Test certificate issuance by verifying the node has received valid certificates. Check that the node can communicate with the cluster's certificate authority. Verify automatic certificate rotation is configured correctly.
Monitor the new node's logs for errors or warnings. Initial integration might reveal configuration issues that weren't apparent during the join process itself.
Removing and Re-adding Nodes
Sometimes you need to remove a node and add it back—perhaps to change its configuration, or to recover from a corrupted state. The process involves removing the node from the cluster, then joining it again as if it were a new node.
When a node is removed, all cluster-related state on that node is cleared. If you then join it again, it receives fresh certificates, a new node ID, and starts with a clean slate. This is essentially adding a new node, not restoring an old one.
Understanding this distinction is important for cluster management. If you need to preserve a node's identity, you must maintain its cluster membership. Removing and re-adding creates a different node.
Scaling Considerations
As you add nodes, be aware of scaling limits. While clusters can grow to hundreds or even thousands of nodes, management node count should remain relatively small. Three to seven management nodes is typical, regardless of total cluster size.
Execution node count scales with workload. Add as many as needed to handle your applications. The limiting factor is usually the cluster's ability to manage all those nodes, not a hard node count limit.
Network bandwidth between nodes becomes a consideration in very large clusters. Ensure your network infrastructure can handle the control plane traffic plus data plane traffic from all nodes.
Geographic Distribution of Joining Nodes
Nodes can join from anywhere network connectivity permits. However, geographic distribution affects cluster performance. Nodes far from management nodes experience higher latency in control plane operations.
For geographically distributed clusters, consider the network topology when joining nodes. Nodes in the same datacenter or region as management nodes experience lower latency. Distant nodes work but might have slower response times for cluster operations.
If you're building a globally distributed cluster, join nodes in batches by location. This helps you identify and resolve location-specific networking issues before they affect the entire cluster.
Cluster Discovery and Join
Join commands require explicitly specifying at least one cluster member's address. There's no automatic cluster discovery—you must know where to join. This explicit approach enhances security by preventing nodes from accidentally joining wrong clusters.
In environments with multiple clusters, this explicitness is valuable. Nodes can't mistakenly join the production cluster when they should join staging. You must explicitly provide the correct cluster address and token.
Some organizations run internal services that provide join information to authorized nodes. These services act as registries, providing the current cluster addresses and tokens to nodes that authenticate successfully. This centralizes join credential management while maintaining security.
Orchestrated Mass Joining
When building large clusters, you might need to join many nodes simultaneously. While management nodes should be added serially, execution nodes can be joined in parallel. The cluster can handle multiple simultaneous execution node joins without issue.
Automation tools facilitate mass joins. Configuration management systems, infrastructure as code tools, and cloud orchestration platforms can all join multiple nodes simultaneously. These tools distribute the join command to many nodes and execute it in parallel.
Monitor cluster health during mass joins. While the system handles many simultaneous joins, there are practical limits. Joining hundreds of nodes simultaneously might overwhelm the cluster. Consider batching—join 50 nodes, verify they're healthy, then join the next batch.
Certificate Authority and Trust
During the join process, the joining node must trust the cluster's certificate authority (CA). This trust is established through the join token and the initial TLS connection. The token proves authorization; the TLS handshake establishes trust in the cluster's CA.
Once trusted, the CA issues certificates to the joining node. These certificates authenticate the node to other cluster members and encrypt communications. The CA's root certificate is distributed to all nodes, creating a web of trust.
Understanding this trust model helps troubleshoot certificate-related join failures. If certificate validation fails, it's usually because time synchronization is off, the CA has changed, or network issues are interfering with certificate issuance.
Join Command Idempotency
The join command is not idempotent. If a node is already part of a cluster, attempting to join again fails with an error. The node must first leave the cluster before it can join (whether joining the same cluster or a different one).
This prevents accidentally disconnecting a node from its cluster. If the join command were idempotent, you could mistakenly move a node to a different cluster by running a join command. The non-idempotent behavior protects against this.
If you need to change a node's cluster membership, explicitly leave the current cluster first, then join the new cluster. This two-step process makes cluster changes intentional and visible.