Observability is critical for operating containerized applications in production. Without proper monitoring and logging, you're flying blind—unable to detect performance degradation, diagnose failures, or understand system behavior. Docker Swarm environments present unique challenges for observability: containers are ephemeral, tasks move between nodes, and multiple replicas of the same service run simultaneously. This article explores how to implement comprehensive monitoring and logging using Prometheus for metrics collection, FluentD for log aggregation, and GELF (Graylog Extended Log Format) for structured logging.
Understanding Swarm Observability Challenges
Container Ephemerality
Containers start, stop, and restart frequently. Traditional monitoring approaches that assume stable, long-lived processes struggle with this dynamism. Your monitoring solution must:
- Automatically discover new containers
- Handle containers that disappear without warning
- Aggregate metrics and logs from multiple replicas
- Maintain historical data even after containers terminate
Distributed Nature
Services run across multiple nodes, creating distributed data sources. Effective observability requires:
- Centralized collection from all nodes
- Correlation of logs and metrics across the cluster
- Understanding of inter-service communication patterns
- Node-level and cluster-level visibility
Multiple Replicas
Services typically run multiple replicas for availability. This creates challenges:
- Distinguishing logs from different replicas
- Aggregating metrics across replicas
- Identifying which replica generated specific events
- Understanding per-replica vs. aggregate performance
Prometheus: Metrics Collection and Monitoring
What is Prometheus?
Prometheus is an open-source monitoring system designed for dynamic, cloud-native environments. It collects numeric time-series data, stores it efficiently, and provides a powerful query language for analysis and alerting.
Prometheus uses a pull-based model: it scrapes metrics from HTTP endpoints exposed by target applications at regular intervals. This approach works well with container orchestration since Prometheus can dynamically discover targets through service discovery mechanisms.
Prometheus Architecture for Swarm
A typical Prometheus deployment in Swarm includes:
Prometheus Server: Scrapes metrics, stores time-series data, and evaluates alerting rules Node Exporter: Exposes hardware and OS metrics from each node cAdvisor: Exposes container resource usage metrics Application Exporters: Expose application-specific metrics Alertmanager: Handles alerts generated by Prometheus
Deploying Prometheus as a Service
Deploy Prometheus as a global service to ensure visibility across the cluster:
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
command:
- '--config.file=/etc/prometheus/prometheus.yml'
- '--storage.tsdb.path=/prometheus'
- '--storage.tsdb.retention.time=30d'
volumes:
- prometheus-data:/prometheus
- prometheus-config:/etc/prometheus
ports:
- "9090:9090"
deploy:
mode: replicated
replicas: 1
placement:
constraints:
- node.role == manager
volumes:
prometheus-data:
prometheus-config:
This configuration deploys Prometheus with 30 days of metric retention and persistent storage.
Prometheus Configuration for Swarm
Configure Prometheus to scrape Swarm services using DNS-based service discovery:
global:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: 'swarm-nodes'
dns_sd_configs:
- names:
- 'tasks.node-exporter'
type: 'A'
port: 9100
- job_name: 'swarm-containers'
dns_sd_configs:
- names:
- 'tasks.cadvisor'
type: 'A'
port: 8080
- job_name: 'application-metrics'
dns_sd_configs:
- names:
- 'tasks.myapp'
type: 'A'
port: 8080
metrics_path: '/metrics'
This configuration discovers targets automatically as services scale up or down.
Deploying Node Exporter
Node Exporter collects host-level metrics. Deploy it as a global service to run on every node:
node-exporter:
image: prom/node-exporter:latest
command:
- '--path.rootfs=/host'
volumes:
- '/:/host:ro,rslave'
deploy:
mode: global
networks:
- monitoring
Global mode ensures every node exposes its metrics, providing complete cluster visibility.
Deploying cAdvisor
cAdvisor exposes container resource usage metrics:
cadvisor:
image: gcr.io/cadvisor/cadvisor:latest
command:
- '--docker_only=true'
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
- /:/rootfs:ro
- /var/run:/var/run:ro
- /sys:/sys:ro
- /var/lib/docker/:/var/lib/docker:ro
deploy:
mode: global
networks:
- monitoring
cAdvisor provides detailed container CPU, memory, network, and disk metrics.
Service Discovery with DNS
Prometheus discovers Swarm services through DNS queries. When you create a service named myapp with 3 replicas, Swarm creates DNS records:
- tasks.myapp resolves to all task IP addresses
- Prometheus queries this DNS name and scrapes each discovered endpoint
This automatic discovery eliminates manual configuration as services scale.
Key Metrics to Monitor
Node-Level Metrics:
- CPU usage and load average
- Memory utilization and available memory
- Disk I/O and space usage
- Network throughput and errors
Container-Level Metrics:
- Per-container CPU usage
- Per-container memory consumption
- Container network traffic
- Container filesystem usage
Application-Level Metrics:
- Request rates and latencies
- Error rates and types
- Business metrics (orders, signups, etc.)
- Custom application-specific metrics
Instrumenting Applications for Prometheus
Applications should expose metrics at a /metrics endpoint. Using the Prometheus client library:
Python Example:
from prometheus_client import Counter, Histogram, start_http_server
import time
# Define metrics
requests_total = Counter('http_requests_total', 'Total HTTP requests')
request_duration = Histogram('http_request_duration_seconds', 'HTTP request duration')
# Instrument your code
@request_duration.time()
def handle_request():
requests_total.inc()
# Your application logic
time.sleep(0.1)
# Expose metrics endpoint
start_http_server(8080)
Go Example:
package main
import (
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
"net/http"
)
var (
requestsTotal = prometheus.NewCounter(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "Total HTTP requests",
},
)
)
func init() {
prometheus.MustRegister(requestsTotal)
}
func handler(w http.ResponseWriter, r *http.Request) {
requestsTotal.Inc()
// Your application logic
}
func main() {
http.Handle("/metrics", promhttp.Handler())
http.HandleFunc("/", handler)
http.ListenAndServe(":8080", nil)
}
Querying Metrics with PromQL
Prometheus Query Language (PromQL) enables powerful metric analysis:
CPU usage per container:
rate(container_cpu_usage_seconds_total[5m])
Memory usage percentage:
(container_memory_usage_bytes / container_spec_memory_limit_bytes) * 100
Request rate by service:
sum(rate(http_requests_total[5m])) by (service)
95th percentile latency:
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
Available disk space:
node_filesystem_avail_bytes / node_filesystem_size_bytes * 100
Setting Up Alerting Rules
Define alerting rules to notify when conditions are met:
groups:
- name: container-alerts
interval: 30s
rules:
- alert: HighCPUUsage
expr: rate(container_cpu_usage_seconds_total[5m]) > 0.8
for: 5m
labels:
severity: warning
annotations:
summary: "High CPU usage detected"
description: "Container {{ $labels.container }} CPU usage is {{ $value }}"
- alert: HighMemoryUsage
expr: (container_memory_usage_bytes / container_spec_memory_limit_bytes) > 0.9
for: 5m
labels:
severity: critical
annotations:
summary: "High memory usage detected"
description: "Container {{ $labels.container }} memory usage is {{ $value }}%"
- alert: ContainerDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Container is down"
description: "Container {{ $labels.instance }} is down"
Deploying Alertmanager
Alertmanager handles alert routing and notification:
alertmanager:
image: prom/alertmanager:latest
command:
- '--config.file=/etc/alertmanager/alertmanager.yml'
volumes:
- alertmanager-config:/etc/alertmanager
ports:
- "9093:9093"
deploy:
mode: replicated
replicas: 1
networks:
- monitoring
Alertmanager Configuration:
global:
resolve_timeout: 5m
route:
group_by: ['alertname', 'cluster']
group_wait: 10s
group_interval: 10s
repeat_interval: 12h
receiver: 'email'
receivers:
- name: 'email'
email_configs:
- to: 'ops@example.com'
from: 'alertmanager@example.com'
smarthost: 'smtp.example.com:587'
auth_username: 'alertmanager'
auth_password: 'password'
FluentD: Log Aggregation
What is FluentD?
FluentD is an open-source data collector that unifies log collection and consumption. It acts as a unified logging layer, collecting logs from multiple sources, processing them, and forwarding them to various destinations.
FluentD's plugin architecture supports hundreds of input and output sources, making it highly versatile for heterogeneous environments.
FluentD Architecture for Swarm
A typical FluentD deployment uses:
FluentD Forwarders: Run on each node to collect local logs FluentD Aggregators: Receive logs from forwarders and route to storage Storage Backend: Elasticsearch, S3, or other log storage systems
Deploying FluentD Forwarders
Deploy FluentD as a global service to collect logs from all nodes:
fluentd:
image: fluent/fluentd:latest
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- fluentd-config:/fluentd/etc
environment:
- FLUENTD_CONF=fluent.conf
deploy:
mode: global
networks:
- logging
FluentD Configuration for Container Logs
Configure FluentD to collect Docker container logs:
<source>
@type tail
path /var/lib/docker/containers/*/*-json.log
pos_file /var/log/fluentd-docker.pos
tag docker.*
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</source>
<filter docker.**>
@type parser
key_name log
<parse>
@type json
</parse>
</filter>
<filter docker.**>
@type record_transformer
<record>
hostname "#{Socket.gethostname}"
tag ${tag}
</record>
</filter>
<match docker.**>
@type forward
<server>
host fluentd-aggregator
port 24224
</server>
<buffer>
@type file
path /var/log/fluentd-buffers/docker
flush_interval 10s
retry_max_interval 30s
</buffer>
</match>
This configuration:
- Tails Docker container log files
- Parses JSON log entries
- Adds hostname and tag metadata
- Forwards logs to aggregator with buffering
Extracting Container Metadata
Enhance logs with container and service information:
<filter docker.**>
@type record_transformer
enable_ruby true
<record>
container_id ${record["container_id"]}
container_name ${record["container_name"]}
service_name ${record["com.docker.swarm.service.name"]}
task_name ${record["com.docker.swarm.task.name"]}
node_id ${record["com.docker.swarm.node.id"]}
</record>
</filter>
This metadata makes logs searchable by service, task, and node.
Deploying FluentD Aggregator
The aggregator receives logs from forwarders and routes them to storage:
fluentd-aggregator:
image: fluent/fluentd:latest
volumes:
- fluentd-aggregator-config:/fluentd/etc
ports:
- "24224:24224"
deploy:
mode: replicated
replicas: 1
networks:
- logging
Aggregator Configuration:
<source>
@type forward
port 24224
bind 0.0.0.0
</source>
<match docker.**>
@type elasticsearch
host elasticsearch
port 9200
logstash_format true
logstash_prefix fluentd
include_tag_key true
tag_key @log_name
<buffer>
flush_interval 10s
retry_max_interval 30s
</buffer>
</match>
Log Parsing and Structuring
Parse unstructured logs into structured fields:
<filter docker.**>
@type parser
key_name log
reserve_data true
<parse>
@type regexp
expression /^(?<timestamp>\S+) (?<level>\S+) (?<message>.*)$/
</parse>
</filter>
This extracts timestamp, log level, and message from log lines.
Multi-Line Log Handling
Handle stack traces and multi-line logs:
<source>
@type tail
path /var/lib/docker/containers/*/*-json.log
pos_file /var/log/fluentd-docker.pos
tag docker.*
format json
<parse>
@type multiline
format_firstline /^\d{4}-\d{2}-\d{2}/
format1 /^(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}) (?<level>[^ ]*) (?<message>.*)/
</parse>
</source>
This groups multi-line stack traces into single log entries.
Log Filtering and Routing
Route logs to different destinations based on content:
<match docker.container.**>
@type rewrite_tag_filter
<rule>
key level
pattern /^ERROR$/
tag error.${tag}
</rule>
<rule>
key level
pattern /^WARN$/
tag warn.${tag}
</rule>
<rule>
key level
pattern /.*/
tag info.${tag}
</rule>
</match>
<match error.**>
@type elasticsearch
host elasticsearch
port 9200
index_name error-logs
</match>
<match warn.**>
@type elasticsearch
host elasticsearch
port 9200
index_name warning-logs
</match>
<match info.**>
@type elasticsearch
host elasticsearch
port 9200
index_name info-logs
</match>
This separates logs by severity into different indices.
FluentD Performance Tuning
Optimize FluentD for high-volume logging:
Buffer Configuration:
<buffer> @type file path /var/log/fluentd-buffers/docker flush_mode interval flush_interval 10s flush_thread_count 4 chunk_limit_size 5M queue_limit_length 32 retry_max_interval 30s retry_forever true </buffer>
Worker Configuration:
<system> workers 4 root_dir /var/log/fluentd </system>
These settings handle high log volumes without dropping messages.
GELF: Graylog Extended Log Format
What is GELF?
GELF is a structured log format designed for efficient log transmission and storage. Unlike plain text logs, GELF embeds metadata directly in log messages, enabling rich searching and filtering without parsing.
GELF supports two transport protocols:
- GELF UDP: Fast, connectionless transmission
- GELF TCP: Reliable, connection-oriented transmission
Docker GELF Logging Driver
Docker includes a native GELF logging driver that sends container logs directly to a GELF endpoint:
services:
myapp:
image: myapp:latest
logging:
driver: gelf
options:
gelf-address: "udp://graylog:12201"
tag: "myapp"
labels: "service,environment"
env: "ENVIRONMENT,VERSION"
This configuration sends all container output to the GELF endpoint with additional metadata.
GELF Message Format
GELF messages are JSON documents with required and optional fields:
Required Fields:
{
"version": "1.1",
"host": "worker-01",
"short_message": "Application started successfully",
"timestamp": 1672531200.123
}
Extended Fields:
{
"version": "1.1",
"host": "worker-01",
"short_message": "HTTP request processed",
"timestamp": 1672531200.123,
"level": 6,
"_service": "web-api",
"_method": "GET",
"_path": "/api/users",
"_status_code": 200,
"_duration_ms": 45
}
Custom fields prefixed with underscore contain application-specific data.
Deploying Graylog
Deploy Graylog as the GELF log collector:
version: '3.8'
services:
mongodb:
image: mongo:4.4
volumes:
- mongo-data:/data/db
networks:
- logging
elasticsearch:
image: elasticsearch:7.17.0
environment:
- discovery.type=single-node
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
volumes:
- es-data:/usr/share/elasticsearch/data
networks:
- logging
graylog:
image: graylog/graylog:4.3
environment:
- GRAYLOG_HTTP_EXTERNAL_URI=http://graylog:9000/
- GRAYLOG_ELASTICSEARCH_HOSTS=http://elasticsearch:9200
- GRAYLOG_MONGODB_URI=mongodb://mongodb:27017/graylog
- GRAYLOG_ROOT_PASSWORD_SHA2=<your-sha2-hash>
ports:
- "9000:9000" # Web interface
- "12201:12201/udp" # GELF UDP
- "12201:12201/tcp" # GELF TCP
volumes:
- graylog-data:/usr/share/graylog/data
depends_on:
- mongodb
- elasticsearch
networks:
- logging
volumes:
mongo-data:
es-data:
graylog-data:
networks:
logging:
Configuring Services with GELF Logging
Configure individual services to use GELF:
web-api:
image: web-api:latest
logging:
driver: gelf
options:
gelf-address: "udp://graylog:12201"
tag: "web-api"
gelf-compression-type: "gzip"
GELF Driver Options:
- gelf-address: GELF endpoint (udp:// or tcp://)
- tag: Identifier for log source
- gelf-compression-type: Compression (gzip or none)
- labels: Docker labels to include
- env: Environment variables to include
GELF with Custom Fields
Include custom fields from environment variables and labels:
database:
image: postgres:14
environment:
- ENVIRONMENT=production
- DATACENTER=us-east
- VERSION=14.5
labels:
service: "postgres"
tier: "database"
logging:
driver: gelf
options:
gelf-address: "udp://graylog:12201"
tag: "postgres"
env: "ENVIRONMENT,DATACENTER,VERSION"
labels: "service,tier"
These fields appear as _environment, _datacenter, _version, _service, and _tier in GELF messages.
Application-Level GELF Logging
Applications can send GELF messages directly using GELF libraries:
Python with pygelf:
import logging
from pygelf import GelfUdpHandler
logger = logging.getLogger()
logger.addHandler(GelfUdpHandler(host='graylog', port=12201))
logger.setLevel(logging.INFO)
logger.info('Application started', extra={
'_service': 'myapp',
'_version': '1.2.3',
'_environment': 'production'
})
Node.js with gelf-pro:
const gelfPro = require('gelf-pro');
gelfPro.setConfig({
host: 'graylog',
port: 12201,
facility: 'myapp'
});
gelfPro.info('Application started', {
service: 'myapp',
version: '1.2.3',
environment: 'production'
});
GELF UDP vs TCP
UDP Advantages:
- Lower latency
- No connection overhead
- Better performance under high load
UDP Disadvantages:
- No delivery guarantee
- Message size limited to ~8KB
- No backpressure mechanism
TCP Advantages:
- Guaranteed delivery
- Supports large messages
- Backpressure prevents overwhelming receiver
TCP Disadvantages:
- Higher latency
- Connection management overhead
- Can block application if receiver is slow
Choose UDP for high-volume, non-critical logs and TCP for critical audit logs.
Searching GELF Logs in Graylog
Graylog provides powerful search capabilities for GELF logs:
Search by service:
_service:web-api
Search by log level:
level:3
Search with time range:
_service:web-api AND timestamp:[now-1h TO now]
Search with field ranges:
_duration_ms:>1000 AND _status_code:500
Complex queries:
_service:web-api AND (_status_code:>=500 OR _duration_ms:>2000)
GELF Log Aggregation Patterns
Per-Service Streams: Create Graylog streams that route logs by service:
- Stream: "Web API Logs" - Rule: _service:web-api
- Stream: "Database Logs" - Rule: _service:postgres
- Stream: "Cache Logs" - Rule: _service:redis
Error Streams: Create streams for different severity levels:
- Stream: "Critical Errors" - Rule: level:<=2
- Stream: "Warnings" - Rule: level:4
- Stream: "Info" - Rule: level:>=6
Business Logic Streams: Create streams for business events:
- Stream: "Payment Processing" - Rule: _event_type:payment
- Stream: "User Authentication" - Rule: _event_type:auth
- Stream: "API Errors" - Rule: _status_code:>=400
Integrating Prometheus and FluentD
Unified Observability Stack
Combine Prometheus metrics with FluentD logs for complete observability:
version: '3.8'
services:
prometheus:
image: prom/prometheus:latest
volumes:
- prometheus-config:/etc/prometheus
- prometheus-data:/prometheus
ports:
- "9090:9090"
networks:
- monitoring
fluentd-aggregator:
image: fluent/fluentd:latest
volumes:
- fluentd-config:/fluentd/etc
ports:
- "24224:24224"
networks:
- logging
grafana:
image: grafana/grafana:latest
environment:
- GF_SECURITY_ADMIN_PASSWORD=admin
ports:
- "3000:3000"
volumes:
- grafana-data:/var/lib/grafana
networks:
- monitoring
- logging
node-exporter:
image: prom/node-exporter:latest
deploy:
mode: global
volumes:
- '/:/host:ro,rslave'
command:
- '--path.rootfs=/host'
networks:
- monitoring
fluentd:
image: fluent/fluentd:latest
deploy:
mode: global
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
- fluentd-forwarder-config:/fluentd/etc
networks:
- logging
volumes:
prometheus-config:
prometheus-data:
fluentd-config:
fluentd-forwarder-config:
grafana-data:
networks:
monitoring:
logging:
Correlating Metrics and Logs
Use consistent labeling to correlate metrics and logs:
Service Configuration:
myapp:
image: myapp:latest
labels:
prometheus.scrape: "true"
prometheus.port: "8080"
prometheus.path: "/metrics"
logging:
driver: gelf
options:
gelf-address: "udp://graylog:12201"
tag: "myapp"
labels: "service,version"
environment:
- SERVICE_NAME=myapp
- SERVICE_VERSION=1.0.0
This allows filtering logs and metrics by the same service identifier.
Grafana Dashboards
Create Grafana dashboards combining metrics and logs:
Metrics Panel - Query Prometheus:
rate(http_requests_total{service="myapp"}[5m])
Logs Panel - Query Elasticsearch (via FluentD):
service:myapp AND level:ERROR
Grafana displays metrics graphs alongside recent error logs from the same service.
Best Practices
Monitoring Best Practices
Use Service Discovery: Configure Prometheus with DNS-based discovery to automatically track services as they scale
Monitor at Multiple Levels: Collect node-level, container-level, and application-level metrics for complete visibility
Set Meaningful Alerts: Alert on symptoms (high error rate) rather than causes (high CPU). Configure appropriate thresholds and durations to avoid alert fatigue
Retain Metrics Appropriately: Store high-resolution metrics for 15-30 days, then downsample or delete older data to manage storage
Label Consistently: Use consistent label names across all metrics (service, environment, version) to enable aggregation
Logging Best Practices
Use Structured Logging: Log in JSON or use GELF to enable rich searching and filtering without parsing
Include Context: Add request IDs, user IDs, and session IDs to correlate logs across services
Separate Log Streams: Use different destinations for application logs, access logs, and audit logs
Control Log Volume: Use log levels appropriately (DEBUG in development, INFO in production) to manage storage costs
Buffer Appropriately: Configure FluentD buffers to handle traffic spikes without dropping logs
GELF Best Practices
Use Custom Fields: Include service metadata (_service, _version, _environment) in every log message
Choose Transport Wisely: Use UDP for high-volume logs, TCP for critical audit logs
Compress When Possible: Enable gzip compression to reduce network usage
Index Strategically: Create separate indices for different log types to optimize search performance
Clean Up Old Logs: Configure retention policies to automatically delete logs after a specified period
Security Considerations
Encrypt Log Transport: Use TLS for FluentD forwarder-to-aggregator communication
Authenticate Endpoints: Require authentication for Prometheus scrape endpoints in production
Sanitize Logs: Ensure passwords, API keys, and sensitive data aren't logged
Control Access: Restrict access to Prometheus, Graylog, and Grafana web interfaces
Audit Log Access: Track who accesses logs, especially for compliance requirements
Docker Swarm observability requires purpose-built tools that handle container ephemerality, distributed architectures, and dynamic scaling. Prometheus provides comprehensive metrics collection with automatic service discovery, FluentD offers flexible log aggregation with extensive plugin support, and GELF enables structured logging with rich metadata. By deploying these tools as Swarm services with appropriate configurations, you gain complete visibility into your containerized applications, enabling faster troubleshooting, proactive monitoring, and data-driven optimization decisions.