Applications require sensitive data like passwords, API keys, and certificates to function. They also need configuration files that define their behavior. Docker Swarm provides dedicated mechanisms for managing both types of data: secrets for sensitive information and configs for non-sensitive configuration data. This article explores how secrets and configs work, how they're encrypted and distributed, and best practices for rotating them safely.
Understanding Docker Secrets
What Are Docker Secrets?
Docker secrets are encrypted blobs of data designed specifically for storing sensitive information. Secrets provide a secure way to make sensitive data available to containers without hardcoding it in images or passing it through environment variables.
When you create a secret, Swarm encrypts it and stores it in the distributed cluster state. Only containers explicitly granted access can read the secret's contents.
Why Secrets Matter
Sensitive data faces several security challenges:
Environment Variables: Visible in container inspect output and process listings Configuration Files in Images: Baked into image layers, accessible to anyone with the image Mounted Files: Require external storage configuration and access management Plain Text Files: Vulnerable if host filesystem is compromised
Secrets solve these problems through encryption at rest, encrypted transmission, and controlled access.
Secret Storage and Encryption
Secrets are stored encrypted in the cluster's distributed state using AES-256-GCM encryption. The encryption keys are managed automatically by Swarm and never leave the cluster.
When managers store secrets:
- Secret content is encrypted with the cluster's encryption key
- Encrypted secret is stored in the distributed database
- Secret metadata (name, labels) is not encrypted for queryability
- Encryption keys are themselves encrypted with additional key encryption keys
Secret Distribution
When a service needs a secret:
- The service definition specifies which secrets to use
- The scheduler assigns tasks to nodes
- Manager nodes send encrypted secrets to nodes running tasks
- Secrets are decrypted only in memory on the destination node
- Secrets are mounted into container filesystems as tmpfs volumes
- When containers stop, secrets are removed from memory
Secrets never touch disk in unencrypted form on worker nodes.
Creating and Managing Secrets
Creating Secrets from Standard Input
Create a secret by piping content:
echo "mypassword123" | docker secret create db_password -
The trailing dash (-) tells Docker to read from standard input.
Creating Secrets from Files
Create a secret from an existing file:
docker secret create db_password ./password.txt
The file content becomes the secret value. The filename is not stored—only the content.
Creating Secrets with Labels
Add metadata labels to secrets:
echo "apikey123" | docker secret create api_key - \ --label environment=production \ --label application=payment-processor
Labels help organize and filter secrets.
Viewing Available Secrets
List all secrets in the cluster:
docker secret ls
This shows secret names, creation times, and update times. Secret values are never displayed.
Inspecting Secret Metadata
View secret details:
docker secret inspect db_password
The inspect output includes:
- Secret ID and name
- Creation and update timestamps
- Labels
- Driver information
The actual secret value is not included in inspect output.
Removing Secrets
Delete a secret:
docker secret rm db_password
You cannot remove secrets currently in use by services. Remove the secret from all services first.
Using Secrets in Services
Attaching Secrets to Services
Grant a service access to secrets:
docker service create \ --name myapp \ --secret db_password \ --secret api_key \ myapp-image
Inside containers, secrets appear as files in /run/secrets/:
/run/secrets/db_password /run/secrets/api_key
Custom Secret Paths
Specify custom filenames for secrets:
docker service create \ --name myapp \ --secret source=db_password,target=database_pwd \ --secret source=api_key,target=keys/api.key \ myapp-image
This mounts secrets at:
/run/secrets/database_pwd /run/secrets/keys/api.key
Secret File Permissions
Secrets are mounted with restrictive permissions:
-r--r----- 1 root root 14 Jan 15 10:30 db_password
Only root and the container's user can read secret files. The files are read-only—containers cannot modify secrets.
Secret File Ownership
Set custom user and group ownership:
docker service create \ --name myapp \ --secret source=db_password,target=db_pwd,uid=1000,gid=1000 \ myapp-image
This makes the secret file owned by user ID 1000 and group ID 1000 instead of root.
Secret File Modes
Customize file permissions:
docker service create \ --name myapp \ --secret source=db_password,mode=0400 \ myapp-image
The mode 0400 makes the file readable only by the owner.
Reading Secrets in Applications
Reading from Files
Applications read secrets as regular files:
Python Example:
with open('/run/secrets/db_password', 'r') as f:
password = f.read().strip()
# Use password for database connection
Node.js Example:
const fs = require('fs');
const password = fs.readFileSync('/run/secrets/db_password', 'utf8').trim();
// Use password in application
Go Example:
password, err := ioutil.ReadFile("/run/secrets/db_password")
if err != nil {
log.Fatal(err)
}
// Use password
Error Handling
Always handle missing secrets gracefully:
import os
secret_path = '/run/secrets/api_key'
if os.path.exists(secret_path):
with open(secret_path, 'r') as f:
api_key = f.read().strip()
else:
# Fall back to environment variable or fail gracefully
api_key = os.getenv('API_KEY')
This allows testing with environment variables during development while using secrets in production.
Secret Rotation
Why Rotate Secrets?
Secret rotation is critical for security:
- Compromised credentials must be changed quickly
- Regular rotation limits exposure windows
- Compliance requirements often mandate periodic rotation
- Employee departures require access revocation
The Secret Update Challenge
Docker secrets are immutable—once created, their values cannot be changed. To rotate a secret:
- Create a new secret with updated value
- Update services to use the new secret
- Remove the old secret
Basic Secret Rotation Process
Rotate a database password:
# Create new secret with updated password echo "newpassword456" | docker secret create db_password_v2 - # Update service to use new secret docker service update \ --secret-rm db_password \ --secret-add db_password_v2 \ myapp # Verify service is using new secret docker service inspect myapp # Remove old secret docker secret rm db_password
Service Update Behavior During Rotation
When you update secrets on a service, Swarm restarts tasks to mount the new secrets. The update follows the service's update configuration:
docker service create \ --name myapp \ --secret db_password \ --update-parallelism 1 \ --update-delay 10s \ myapp-image
During secret rotation, tasks are updated one at a time with 10-second delays between updates.
Zero-Downtime Secret Rotation
For truly zero-downtime rotation, add the new secret before removing the old one:
# Add new secret alongside old docker service update \ --secret-add db_password_v2 \ myapp # Application now has access to both: # /run/secrets/db_password (old) # /run/secrets/db_password_v2 (new) # Application code reads new secret # After verification, remove old secret docker service update \ --secret-rm db_password \ myapp
This requires application logic to handle multiple secret versions.
Automated Secret Rotation
Script secret rotation for regular execution:
#!/bin/bash
SERVICE_NAME="myapp"
SECRET_NAME="api_key"
NEW_SECRET_NAME="${SECRET_NAME}_$(date +%s)"
# Generate new secret value
NEW_VALUE=$(openssl rand -base64 32)
# Create new secret
echo "$NEW_VALUE" | docker secret create "$NEW_SECRET_NAME" -
# Update service
docker service update \
--secret-rm "$SECRET_NAME" \
--secret-add "source=${NEW_SECRET_NAME},target=${SECRET_NAME}" \
"$SERVICE_NAME"
# Wait for service update to complete
sleep 30
# Remove old secrets (keep last 2 versions)
docker secret ls --format "{{.Name}}" | \
grep "^${SECRET_NAME}_" | \
sort -r | \
tail -n +3 | \
xargs -r docker secret rm
Run this script on a schedule for automated rotation.
Secret Rotation Best Practices
Version Secret Names: Include version numbers or timestamps in secret names:
db_password_v1 db_password_v2 api_key_20250115 api_key_20250215
Use Target Names: When adding versioned secrets, use target to maintain consistent paths:
docker service update \ --secret-add source=db_password_v2,target=db_password \ myapp
Keep Old Versions Temporarily: Maintain the previous secret version for quick rollback:
# Keep v1 and v2, remove older versions docker secret ls | grep db_password
Document Rotation Schedule: Track when secrets were last rotated and when next rotation is due.
Test Rotation Process: Regularly test rotation procedures in non-production environments.
Understanding Docker Configs
What Are Docker Configs?
Configs are similar to secrets but designed for non-sensitive data like application configuration files, web server configs, or service parameters. Unlike secrets, configs are not encrypted—they're stored as plain text in the cluster state.
Configs provide:
- Centralized configuration management
- Version control for configuration files
- Dynamic configuration updates without rebuilding images
- Separation of configuration from application code
When to Use Configs vs Secrets
Use Secrets For:
- Passwords and passphrases
- API keys and tokens
- TLS certificates and private keys
- Database connection strings with credentials
- OAuth secrets
- Any sensitive data
Use Configs For:
- Application configuration files
- Web server configurations (nginx.conf, apache.conf)
- Logging configurations
- Feature flags and application settings
- Public certificates (not private keys)
- Non-sensitive environment-specific settings
Configs vs Environment Variables
Configs are preferable to environment variables when:
- Configuration data is complex or multi-line
- Configuration includes special characters
- Configuration needs version control
- Configuration is shared across multiple services
- Configuration size exceeds environment variable limits
Creating and Managing Configs
Creating Configs from Standard Input
Create a config by piping content:
cat <<EOF | docker config create nginx_config -
server {
listen 80;
server_name example.com;
location / {
proxy_pass http://backend;
}
}
EOF
Creating Configs from Files
Create a config from an existing file:
docker config create app_settings ./config.yaml
Creating Configs with Labels
Add metadata labels:
docker config create app_config ./app.conf \ --label version=1.0 \ --label environment=production
Viewing Available Configs
List all configs:
docker config ls
Shows config names, creation dates, and update dates.
Inspecting Config Metadata
View config details:
docker config inspect nginx_config
The inspect output includes the actual config data (unlike secrets which hide values).
Removing Configs
Delete a config:
docker config rm nginx_config
Cannot remove configs currently used by services.
Using Configs in Services
Attaching Configs to Services
Mount configs into containers:
docker service create \ --name web \ --config source=nginx_config,target=/etc/nginx/nginx.conf \ nginx
The config file appears at /etc/nginx/nginx.conf inside containers.
Multiple Configs
Attach multiple configs to a service:
docker service create \ --name app \ --config source=app_config,target=/app/config.yaml \ --config source=db_config,target=/app/database.conf \ --config source=logging_config,target=/app/logging.conf \ myapp-image
Config File Permissions
Specify custom permissions:
docker service create \ --name app \ --config source=app_config,target=/app/config.yaml,mode=0644 \ myapp-image
Config File Ownership
Set custom user and group ownership:
docker service create \ --name app \ --config source=app_config,target=/app/config.yaml,uid=1000,gid=1000 \ myapp-image
Config Rotation
The Config Update Process
Like secrets, configs are immutable. To update a configuration:
# Create new version cat new_config.yaml | docker config create app_config_v2 - # Update service docker service update \ --config-rm app_config \ --config-add source=app_config_v2,target=/app/config.yaml \ myapp # Remove old config after verification docker config rm app_config
Config Update Without Service Restart
Some applications can reload configuration without restarting. However, Swarm still restarts containers when config mounts change. To avoid restarts:
- Use configs for initial bootstrap configuration
- Implement dynamic configuration loading from external sources
- Use signals to reload configuration without Swarm updates
Versioned Config Pattern
Maintain multiple config versions:
# Create versioned configs docker config create app_config_v1 ./config_v1.yaml docker config create app_config_v2 ./config_v2.yaml docker config create app_config_v3 ./config_v3.yaml # Services use specific versions docker service create \ --name app \ --config source=app_config_v2,target=/app/config.yaml \ myapp-image
Rolling Back Configs
Rollback to a previous config version:
# Currently using v3, rollback to v2 docker service update \ --config-rm app_config_v3 \ --config-add source=app_config_v2,target=/app/config.yaml \ myapp
The service restarts with the previous configuration.
Combining Secrets and Configs
Services with Both Secrets and Configs
Use both mechanisms in a single service:
docker service create \ --name webapp \ --secret db_password \ --secret api_key \ --config source=app_config,target=/app/config.yaml \ --config source=nginx_config,target=/etc/nginx/nginx.conf \ webapp-image
Inside containers:
# Secrets in tmpfs /run/secrets/db_password /run/secrets/api_key # Configs as regular files /app/config.yaml /etc/nginx/nginx.conf
Configuration File Referencing Secrets
Configuration files can reference secret paths:
# app_config.yaml database: password_file: /run/secrets/db_password connection_string: "postgresql://user@host/db" api: key_file: /run/secrets/api_key endpoint: "https://api.example.com"
Applications read the config file to find secret paths, then read secrets from those paths.
Template Pattern
Create config templates that reference environment variables pointing to secrets:
# Config with environment variable placeholders
cat <<EOF | docker config create app_template -
{
"db_password": "${DB_PASSWORD}",
"api_key": "${API_KEY}"
}
EOF
# Start service with initialization script
docker service create \
--name app \
--secret db_password \
--secret api_key \
--config app_template \
--entrypoint /init.sh \
myapp-image
The init script reads secrets and substitutes them into the config template.
Secret and Config Encryption
Encryption at Rest
Secrets are encrypted in the cluster state using AES-256-GCM. The encryption process:
- Swarm generates a cluster-wide encryption key during initialization
- The encryption key is itself encrypted with a key encryption key (KEK)
- The KEK is distributed to manager nodes
- Secrets are encrypted with the data encryption key before storage
- Encrypted secrets are stored in the distributed database
Configs are stored unencrypted but benefit from access controls.
Encryption in Transit
All communication between Swarm nodes uses mutual TLS:
- Secrets transmitted from managers to workers are encrypted in transit
- Configs transmitted from managers to workers are also encrypted in transit
- Node-to-node traffic uses TLS 1.2 or higher
Autolock Feature
Enable autolock to require manual unlocking of managers after restart:
docker swarm update --autolock=true
After enabling autolock, Swarm displays an unlock key:
SWMKEY-1-...
Store this key securely. After manager restart, unlock with:
docker swarm unlock
Enter the unlock key when prompted.
Autolock protects against scenarios where an attacker gains access to a manager's filesystem. Without the unlock key, they cannot decrypt the cluster state or secrets.
Autolock Key Rotation
Rotate the autolock key:
docker swarm unlock-key --rotate
This generates a new key. The old key becomes invalid. Update your secure storage with the new key.
Viewing Current Unlock Key
Retrieve the current unlock key:
docker swarm unlock-key
Use this when you need to share the key with other administrators or backup systems.
Secret and Config Storage Locations
Manager Node Storage
On manager nodes, encrypted secrets and configs are stored in:
/var/lib/docker/swarm/certificates/ /var/lib/docker/swarm/raft/
The Raft directory contains the distributed state including encrypted secrets. Never copy these directories to untrusted locations.
Worker Node Storage
Worker nodes never store secrets persistently. Secrets exist only:
- In memory while tasks are running
- In tmpfs mounts inside containers
When tasks stop, secrets are removed from worker memory.
Backup Considerations
Backing up Swarm state backs up secrets:
# Stop Docker on a manager systemctl stop docker # Backup swarm directory tar czf swarm-backup.tar.gz /var/lib/docker/swarm # Restart Docker systemctl start docker
Store backups securely. They contain encrypted secrets but also the keys to decrypt them.
Access Control for Secrets and Configs
Service-Level Access Control
Access control is service-based. Only services explicitly granted access can read secrets or configs:
# Service 1 has access docker service create \ --name app1 \ --secret db_password \ app1-image # Service 2 does not have access docker service create \ --name app2 \ app2-image
Service 2 cannot access the db_password secret.
No Fine-Grained ACLs
Swarm doesn't support user-based or group-based access control for secrets. Any service can be granted access to any secret by anyone with access to create or update services.
For environments requiring stricter access control, integrate with external secret management systems.
External Secret Management Integration
For advanced secret management, integrate external systems:
HashiCorp Vault: Store secrets in Vault, retrieve at service startup AWS Secrets Manager: Use AWS SDK to fetch secrets at runtime Azure Key Vault: Integrate with Azure services for secret retrieval Google Secret Manager: Fetch secrets during container initialization
These systems provide features like:
- Fine-grained access control
- Audit logging
- Dynamic secret generation
- Centralized secret management across multiple clusters
Secret and Config Naming Strategies
Descriptive Naming
Use clear, descriptive names:
# Good names docker secret create postgres_password_prod - docker secret create api_key_payment_gateway - docker config create nginx_reverse_proxy_config - # Avoid docker secret create pwd1 - docker secret create key - docker config create config1 -
Environment-Based Naming
Include environment in names:
docker secret create db_password_dev - docker secret create db_password_staging - docker secret create db_password_prod -
Service-Based Naming
Prefix with service name:
docker secret create webapp_db_password - docker secret create webapp_session_secret - docker secret create api_jwt_secret -
Version-Based Naming
Include version or timestamp:
docker secret create api_key_v1 - docker secret create api_key_v2 - docker secret create cert_20250115 - docker secret create cert_20250215 -
Troubleshooting Secrets and Configs
Secret Not Available in Container
If a secret doesn't appear in /run/secrets/:
# Verify secret exists
docker secret ls
# Check service has secret attached
docker service inspect myapp --format '{{.Spec.TaskTemplate.ContainerSpec.Secrets}}'
# Verify task is running
docker service ps myapp
# Check container mount points
docker exec container_id mount | grep secrets
Service Won't Start After Secret Rotation
If service fails after secret update:
# Check service logs docker service logs myapp # Inspect service tasks docker service ps myapp --no-trunc # Rollback if needed docker service update --rollback myapp
Common issues:
- Application doesn't handle secret changes
- New secret value is invalid
- File permissions prevent reading
Config File Not Loaded
If application doesn't see config changes:
# Verify config exists
docker config ls
# Check config is attached to service
docker service inspect myapp --format '{{.Spec.TaskTemplate.ContainerSpec.Configs}}'
# Verify config content
docker config inspect myconfig
# Check file exists in container
docker exec container_id ls -la /path/to/config
Permission Denied Errors
If applications can't read secrets or configs:
# Check file permissions in container docker exec container_id ls -la /run/secrets/ docker exec container_id ls -la /path/to/config # Verify user ID running process docker exec container_id ps aux # Update secret/config with correct UID docker service update \ --secret-rm old_secret \ --secret-add source=new_secret,uid=1000 \ myapp
Best Practices for Secrets and Configs
Secret Management Best Practices
Never Log Secrets: Ensure applications don't log secret values Minimize Secret Distribution: Only grant secrets to services that need them Regular Rotation: Rotate secrets on a schedule Version Control: Track secret versions for rollback capability Audit Access: Monitor which services access which secrets Use External Systems: For complex requirements, integrate dedicated secret management platforms
Config Management Best Practices
Version Configs: Maintain versioned configs for rollback Document Changes: Use labels to document config versions and changes Test Before Production: Test config changes in non-production environments Keep Configs Small: Large configs slow service updates Use External Config Stores: For dynamic configuration, use external config services
Rotation Best Practices
Automate Rotation: Script and schedule regular secret rotation Coordinate Rotation: Update all dependent services during rotation Test Rotation Process: Regularly test rotation procedures Plan for Rollback: Maintain previous secret versions temporarily Monitor After Rotation: Verify services function correctly with new secrets
Security Best Practices
Enable Autolock: Protect cluster state with autolock in sensitive environments Backup Encryption Keys: Securely store unlock keys Limit Manager Access: Restrict who can access manager nodes Audit Secret Usage: Track secret creation, updates, and deletions Separate Environments: Use different secrets for dev, staging, and production
Docker Swarm's secrets and configs provide secure, manageable mechanisms for distributing sensitive and non-sensitive data to services. Secrets are encrypted at rest and in transit, distributed only to authorized services, and mounted in memory to prevent disk exposure. Configs offer similar distribution capabilities for non-sensitive data. Both support rotation through versioning patterns, enabling security best practices like regular credential rotation without service downtime. By understanding encryption mechanisms, implementing systematic rotation procedures, and following security best practices, you can build secure, maintainable systems that protect sensitive data throughout its lifecycle.