Containers are ephemeral by design—when a container stops, its writable layer disappears. For stateless applications, this is perfect. However, databases, file stores, and other stateful applications need persistent storage that survives container restarts and can follow tasks as they're rescheduled across nodes. This article explores persistent storage solutions for Docker Swarm, including volume plugins, cross-node storage systems, and patterns for managing stateful workloads in distributed environments.

Understanding Storage Challenges in Swarm

The Container Storage Problem

Containers have three storage layers:

Image Layers: Read-only layers from the container image Container Layer: Thin writable layer created when container starts Volumes: Persistent storage mounted into containers

When a container stops, the container layer is deleted. Any data written there is lost. Volumes provide persistence, but vanilla Docker volumes are local to a single host.

Multi-Node Storage Challenges

In a Swarm cluster, tasks can be scheduled on any node. This creates storage challenges:

Task Mobility: When a task moves to a different node, it needs access to the same data

Data Consistency: Multiple replicas may try to write to the same data simultaneously

Performance: Network-attached storage introduces latency compared to local disks

Availability: Storage systems become critical dependencies

Solving these challenges requires storage solutions that work across the cluster.

Docker Volume Basics

Creating Local Volumes

Create a standard local volume:

docker volume create mydata

This creates a volume on the local node only. Services using this volume must be pinned to this specific node.

Mounting Volumes in Services

Attach volumes to services:

docker service create \
  --name database \
  --mount type=volume,source=mydata,target=/var/lib/mysql \
  mysql:8

The volume is mounted at /var/lib/mysql inside containers.

Volume Mount Options

Specify additional mount options:

docker service create \
  --name app \
  --mount type=volume,source=appdata,target=/data,readonly \
  app-image

The readonly option prevents containers from writing to the volume.

Volume Inspection

View volume details:

docker volume inspect mydata

Shows volume name, driver, mountpoint, and options.

Local Volume Limitations

Local volumes have critical limitations in Swarm:

Exist only on one node
Tasks using local volumes must be constrained to that node
No automatic replication
Node failure means data loss unless separately backed up
Cannot support multi-replica services that need shared data

These limitations make local volumes unsuitable for most distributed scenarios.

Volume Drivers and Plugins

What Are Volume Drivers?

Volume drivers extend Docker's storage capabilities. While the default driver creates local volumes, plugin drivers enable:

Network-attached storage
Cloud storage integration
Distributed filesystems
Storage replication and redundancy

Installing Volume Plugins

Volume plugins are typically distributed as Docker plugins:

# Example: Installing REX-Ray plugin
docker plugin install rexray/ebs \
  EBS_ACCESSKEY=XXX \
  EBS_SECRETKEY=YYY \
  --grant-all-permissions

The --grant-all-permissions grants necessary privileges to the plugin.

Listing Installed Plugins

View available plugins:

docker plugin ls

Shows plugin names, status (enabled/disabled), and capabilities.

Plugin Management

Enable or disable plugins:

# Disable plugin
docker plugin disable rexray/ebs

# Enable plugin
docker plugin enable rexray/ebs

# Remove plugin
docker plugin rm rexray/ebs

Plugins can only be removed when no volumes are using them.

Network-Attached Storage with NFS

NFS Volume Driver

NFS (Network File System) is a widely-supported network storage protocol. Docker includes built-in NFS support through volume driver options.

Setting Up NFS Server

First, configure an NFS server (example on Ubuntu):

# Install NFS server
apt-get install nfs-kernel-server

# Create export directory
mkdir -p /srv/nfs/swarm-data
chown nobody:nogroup /srv/nfs/swarm-data
chmod 777 /srv/nfs/swarm-data

# Configure exports
echo "/srv/nfs/swarm-data *(rw,sync,no_subtree_check,no_root_squash)" >> /etc/exports

# Apply configuration
exportfs -ra
systemctl restart nfs-kernel-server

Creating NFS Volumes

Create a volume using NFS:

docker volume create \
  --driver local \
  --opt type=nfs \
  --opt o=addr=nfs-server.example.com,rw \
  --opt device=:/srv/nfs/swarm-data \
  nfs-volume

This creates a volume that mounts the NFS export.

Using NFS Volumes in Services

Deploy services with NFS volumes:

docker service create \
  --name webapp \
  --replicas 3 \
  --mount type=volume,source=nfs-volume,target=/app/data \
  webapp-image

All three replicas share the same NFS mount, accessing identical data.

NFS Volume Driver Options

Customize NFS mounts with options:

docker volume create \
  --driver local \
  --opt type=nfs \
  --opt o=addr=nfs-server,vers=4.1,rw \
  --opt device=:/export/path \
  nfs-vol-v4

The vers=4.1 specifies NFS version 4.1.

NFS Performance Considerations

NFS introduces network latency:

Sequential Reads/Writes: Reasonable performance for bulk operations Random I/O: Significantly slower than local disks Metadata Operations: Creating/deleting files is slower Multiple Clients: Performance decreases with many concurrent clients

For high-performance requirements, consider alternatives.

NFS High Availability

Make NFS highly available:

Multiple NFS Servers: Deploy redundant NFS servers with shared storage backend Load Balancing: Use DNS or load balancers to distribute clients Failover Clustering: Implement automatic failover between NFS servers

Without HA, NFS server failure stops all dependent services.

REX-Ray for Cloud Storage

What is REX-Ray?

REX-Ray is a volume plugin that integrates Docker with cloud storage services:

AWS EBS (Elastic Block Store)
Google Persistent Disks
Azure Managed Disks
OpenStack Cinder
VirtualBox storage

REX-Ray handles volume creation, attachment, and detachment automatically.

Installing REX-Ray for AWS EBS

Install REX-Ray plugin for AWS:

docker plugin install rexray/ebs \
  EBS_ACCESSKEY=your-access-key \
  EBS_SECRETKEY=your-secret-key \
  EBS_REGION=us-east-1 \
  --grant-all-permissions

Creating EBS Volumes

Create a volume backed by AWS EBS:

docker volume create \
  --driver rexray/ebs \
  --opt size=10 \
  --opt volumetype=gp3 \
  ebs-volume

This creates a 10GB GP3 EBS volume in AWS.

Using REX-Ray Volumes

Deploy services with REX-Ray volumes:

docker service create \
  --name postgres \
  --replicas 1 \
  --mount type=volume,source=ebs-volume,target=/var/lib/postgresql/data \
  postgres:14

When the task is scheduled, REX-Ray:

Attaches the EBS volume to the node running the task
Mounts the volume inside the container
If the task moves, REX-Ray detaches from old node and attaches to new node

REX-Ray Limitations

Single Attachment: EBS volumes can only attach to one node at a time Replica Count: Services using REX-Ray must have --replicas 1 Attachment Time: Moving tasks incurs EBS attach/detach delays (10-30 seconds) Regional Constraint: Volumes and nodes must be in the same AWS region/zone

These limitations make REX-Ray suitable for single-instance stateful services, not multi-replica workloads.

REX-Ray Configuration File

Configure REX-Ray via config file for more control:

# /etc/rexray/config.yml
libstorage:
  service: ebs
ebs:
  accessKey: YOUR_ACCESS_KEY
  secretKey: YOUR_SECRET_KEY
  region: us-east-1

Then install without inline credentials:

docker plugin install rexray/ebs --grant-all-permissions

GlusterFS for Distributed Storage

What is GlusterFS?

GlusterFS is a distributed network filesystem that aggregates storage from multiple servers into a single namespace. It provides:

Scalability across many storage servers
Replication for data redundancy
High availability through redundancy
POSIX-compliant filesystem interface

GlusterFS Architecture

GlusterFS consists of:

Bricks: Storage directories on individual servers Volumes: Logical volumes composed of multiple bricks Gluster Clients: Nodes that mount GlusterFS volumes

Setting Up GlusterFS Cluster

Install GlusterFS on storage nodes:

# On each storage node
apt-get install glusterfs-server
systemctl start glusterd
systemctl enable glusterd

# Peer the nodes (run on one node)
gluster peer probe storage-node-2
gluster peer probe storage-node-3

# Create a replicated volume
gluster volume create swarm-data replica 3 \
  storage-node-1:/data/brick1 \
  storage-node-2:/data/brick1 \
  storage-node-3:/data/brick1 \
  force

# Start the volume
gluster volume start swarm-data

This creates a 3-way replicated volume.

Mounting GlusterFS Volumes

Create Docker volumes backed by GlusterFS:

docker volume create \
  --driver local \
  --opt type=glusterfs \
  --opt o=addr=storage-node-1,backup-volfile-servers=storage-node-2:storage-node-3 \
  --opt device=:/swarm-data \
  gluster-volume

Using GlusterFS Volumes in Services

Deploy services with GlusterFS:

docker service create \
  --name webapp \
  --replicas 5 \
  --mount type=volume,source=gluster-volume,target=/app/uploads \
  webapp-image

All five replicas share the same GlusterFS volume with replicated data.

GlusterFS Volume Types

GlusterFS supports different volume types:

Distributed: Files distributed across bricks (no redundancy) Replicated: Files replicated across bricks (high redundancy) Distributed-Replicated: Combination of both

For Swarm, use replicated or distributed-replicated volumes for data safety.

GlusterFS Performance

GlusterFS performance characteristics:

Reads: Scale with number of storage nodes Writes: Limited by replication factor (writes go to all replicas) Small Files: Metadata overhead impacts performance Large Files: Better performance for bulk operations

Tune GlusterFS parameters for your workload patterns.

Flocker for Container Data Management

What is Flocker?

Flocker is a data volume manager for containerized applications. It provides:

Volume migration between hosts
ZFS-based storage with snapshots
Integration with cloud storage backends
Dataset-centric storage management

Note: Flocker is no longer actively maintained, but similar concepts apply to modern alternatives.

Alternative: Portworx

Portworx is a modern container-native storage platform:

Software-defined storage across cluster nodes
Volume replication and high availability
Snapshots and clones
Disaster recovery features
Cloud-native integration

Install Portworx on Swarm nodes and use its volume driver for persistent storage across the cluster.

Convoy for Volume Snapshots

What is Convoy?

Convoy is a Docker volume plugin supporting:

Volume snapshots
Volume backups
Device Mapper thin provisioning
NFS backend support

Creating Snapshot-Enabled Volumes

With Convoy installed:

docker volume create \
  --driver convoy \
  --opt backup=incremental \
  convoy-volume

Taking Volume Snapshots

Create snapshots of volumes:

# Using Convoy CLI
convoy snapshot create \
  --name snapshot-1 \
  convoy-volume

Snapshots capture point-in-time states for backup or rollback.

Ceph RBD for Block Storage

What is Ceph?

Ceph is a distributed storage system providing:

Object storage
Block storage (RBD)
Filesystem (CephFS)

For Docker, Ceph RBD provides block devices that can be mounted as volumes.

Ceph RBD Volume Driver

Use Ceph RBD with Docker:

docker volume create \
  --driver rbd \
  --opt size=10 \
  rbd-volume

Ceph RBD volumes can be attached to containers, providing persistent block storage backed by Ceph's distributed architecture.

Ceph Replication

Ceph automatically replicates data across cluster nodes. Configure replication level:

2x replication: Data stored on two nodes
3x replication: Data stored on three nodes (recommended)

Higher replication increases availability and durability.

Volume Sharing Patterns

Single-Writer Multiple-Reader

One task writes, many tasks read:

# Writer service
docker service create \
  --name writer \
  --replicas 1 \
  --mount type=volume,source=shared-data,target=/data \
  writer-app

# Reader services
docker service create \
  --name reader \
  --replicas 5 \
  --mount type=volume,source=shared-data,target=/data,readonly \
  reader-app

Readers mount the volume as read-only, preventing accidental writes.

Exclusive Single-Writer

Only one task writes, enforced at scheduling:

docker service create \
  --name database \
  --replicas 1 \
  --constraint 'node.labels.storage-role==primary' \
  --mount type=volume,source=db-data,target=/var/lib/postgresql/data \
  postgres

Constraint ensures only one replica exists, preventing write conflicts.

Coordinated Multi-Writer

Multiple tasks write with coordination:

Use application-level locking or coordination:

Database locking mechanisms
Distributed locks (e.g., etcd, Consul)
Application-level synchronization

The storage system must support concurrent writes safely (like GlusterFS or shared database storage).

Local Volume Constraints

Pinning Services to Nodes

When using local volumes, pin services to specific nodes:

# Label node with storage
docker node update --label-add storage=local-db node-1

# Create service constrained to that node
docker service create \
  --name database \
  --replicas 1 \
  --constraint 'node.labels.storage==local-db' \
  --mount type=volume,source=local-db-vol,target=/var/lib/postgresql/data \
  postgres

The service always runs on node-1 where the local volume exists.

Multiple Local Volumes

Deploy multiple instances with separate local volumes:

# Label nodes
docker node update --label-add db-instance=1 node-1
docker node update --label-add db-instance=2 node-2

# Create constrained services
docker service create \
  --name db-instance-1 \
  --replicas 1 \
  --constraint 'node.labels.db-instance==1' \
  --mount type=volume,source=db-vol-1,target=/var/lib/postgresql/data \
  postgres

docker service create \
  --name db-instance-2 \
  --replicas 1 \
  --constraint 'node.labels.db-instance==2' \
  --mount type=volume,source=db-vol-2,target=/var/lib/postgresql/data \
  postgres

Each instance has its own local volume on its designated node.

Bind Mounts for Host Paths

Creating Bind Mounts

Mount host directories into containers:

docker service create \
  --name app \
  --mount type=bind,source=/host/path,target=/container/path \
  app-image

This directly mounts /host/path from the node into the container.

Bind Mount Use Cases

Bind mounts are useful for:

Accessing host configuration files
Sharing data between host and container
Development environments with code on host

Bind Mount Limitations

Node-Specific: Bind mounts reference local host paths Must Exist: Path must exist on the host No Portability: Services using bind mounts must be constrained to specific nodes

For production, prefer volume plugins over bind mounts.

Bind Mount with Node Constraints

Ensure path exists on target nodes:

# Label nodes with shared mount
docker node update --label-add has-shared-mount=true node-1
docker node update --label-add has-shared-mount=true node-2

# Create service
docker service create \
  --name app \
  --constraint 'node.labels.has-shared-mount==true' \
  --mount type=bind,source=/mnt/shared,target=/data \
  app-image

Only schedules on nodes labeled as having the bind mount path.

Tmpfs Mounts for Memory Storage

Creating Tmpfs Mounts

Store data in memory:

docker service create \
  --name cache \
  --mount type=tmpfs,target=/cache,tmpfs-size=1g \
  cache-app

Tmpfs mounts are stored in RAM, providing extremely fast access but no persistence.

Tmpfs Use Cases

Use tmpfs for:

Temporary caches
Session data
Build artifacts
Any ephemeral data

Tmpfs Limitations

Non-Persistent: Data lost when container stops Memory Consumption: Uses node's RAM Size Limits: Constrained by available memory

Backup and Recovery Strategies

Volume Backup Approaches

Snapshot-Based: Use storage system snapshots (Ceph, Portworx) File-Based: Copy files from mounted volumes Database-Specific: Use database backup tools (pg_dump, mysqldump)

Backup Service Pattern

Create a backup service:

docker service create \
  --name backup \
  --mode global \
  --mount type=volume,source=db-data,target=/backup-source,readonly \
  --mount type=volume,source=backup-dest,target=/backup-dest \
  backup-image

Backup service reads from source volumes and writes to backup destination.

Scheduled Backups

Use cron or systemd timers to schedule backups:

# Example backup script
#!/bin/bash
BACKUP_DATE=$(date +%Y%m%d-%H%M%S)
docker run --rm \
  -v db-data:/source:ro \
  -v /backups:/dest \
  backup-tool \
  backup /source /dest/backup-$BACKUP_DATE

Disaster Recovery

Plan for volume recovery:

Maintain backup of volume data
Document volume creation procedures
Test restore procedures regularly
Keep backup retention policy

Performance Optimization

Choosing the Right Storage Backend

Match storage to workload:

High IOPS Databases: Local SSDs or high-performance cloud block storage Shared File Storage: NFS or GlusterFS for concurrent access Large Media Files: Object storage with CDN Logs and Metrics: Fast sequential write storage

Caching Strategies

Implement caching layers:

Application Cache: Redis or Memcached for hot data CDN: CloudFront or Cloudflare for static assets Local Caching: tmpfs for frequently accessed data

Reduce storage system load through caching.

Storage Tiering

Use different storage tiers:

Hot Data: Fast SSDs or local storage Warm Data: Network storage (NFS, GlusterFS) Cold Data: Object storage (S3, Minio)

Move data between tiers based on access patterns.

Best Practices for Swarm Storage

Volume Naming Conventions

Use descriptive volume names:

# Good names
docker volume create prod-database-data
docker volume create staging-uploads

# Avoid
docker volume create vol1
docker volume create data

Label Volumes

Add metadata labels:

docker volume create \
  --label environment=production \
  --label application=webapp \
  --label created-by=admin \
  prod-webapp-data

Labels help organize and filter volumes.

Regular Backups

Implement automated backup schedules:

Daily backups for critical data
Weekly backups for less critical data
Retain multiple backup generations
Store backups off-cluster

Test Restore Procedures

Regularly test volume restoration:

Restore from backup to test environment
Verify data integrity
Document restore process
Time the restore procedure

Monitor Storage Usage

Track storage metrics:

Volume utilization
IOPS and throughput
Latency percentiles
Error rates

Set alerts for approaching capacity limits.

Plan for Growth

Anticipate storage growth:

Monitor growth trends
Provision ahead of needs
Plan for storage system expansion
Consider storage lifecycle management

Security Considerations

Secure persistent storage:

Encrypt volumes at rest
Encrypt network storage traffic
Implement access controls
Audit storage access

Documentation

Document storage architecture:

Which services use which volumes
Storage backend configurations
Backup and recovery procedures
Troubleshooting guides

Persistent storage in Docker Swarm requires careful planning and the right tools. Local volumes work for development but lack the mobility needed in production clusters. Volume plugins like REX-Ray, distributed filesystems like GlusterFS, and network storage solutions like NFS provide the cross-node accessibility stateful applications require. By understanding the trade-offs between different storage solutions—considering factors like performance, availability, consistency, and operational complexity—you can architect storage systems that meet your applications' durability and accessibility requirements while operating efficiently across your Swarm cluster.

Persistent Storage in Multi-Node Swarm: Plugins and Cross-Node Volumes