Advanced Healthchecks and Dependency Patterns in Docker Compose

Building Robust Service Dependencies and Intelligent Health Monitoring

Tags: #docker-compose #healthchecks #service-dependencies #container-health #startup-order #depends-on #docker

Container orchestration requires more than just starting services—it demands intelligent coordination of service startup, continuous health monitoring, and graceful handling of failures. Docker Compose provides sophisticated mechanisms for defining service dependencies and implementing comprehensive healthchecks that ensure your multi-container applications start reliably, run stably, and recover gracefully from failures.

Understanding Basic Service Dependencies

The depends_on directive establishes startup order relationships between services. When you define dependencies, Docker Compose starts services in the correct sequence:

version: '3.8'

services:
  web:
    image: nginx:alpine
    depends_on:
      - api
    
  api:
    image: node:18-alpine
    depends_on:
      - database
    
  database:
    image: postgres:14

This configuration creates a startup chain: database → api → web. Docker Compose ensures that the database starts before the API, and the API starts before the web server.

However, depends_on only controls when containers are created and started—it doesn't wait for services to be ready. A database container might be running but not yet accepting connections. This is where healthchecks become essential.

Implementing Basic Healthchecks

Healthchecks allow containers to report their operational status. Docker periodically executes a health check command and marks containers as healthy or unhealthy based on the result:

version: '3.8'

services:
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

Healthcheck parameters:

• test: Command to execute for health verification
• interval: Time between health checks (default: 30s)
• timeout: Maximum time for health check execution (default: 30s)
• retries: Consecutive failures before marking unhealthy (default: 3)
• start_period: Grace period during container initialization (default: 0s)

The container status progresses through states:

1. starting: Initial state during start_period
2. healthy: Health check succeeded
3. unhealthy: Health check failed retries consecutive times

Condition-Based Dependencies

Combine depends_on with healthcheck conditions to wait for services to be ready:

version: '3.8'

services:
  web:
    image: nginx:alpine
    depends_on:
      api:
        condition: service_healthy
    
  api:
    image: node:18-alpine
    depends_on:
      database:
        condition: service_healthy
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 40s
    
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
    
  cache:
    image: redis:alpine

Available conditions:

• service_started: Wait for container to start (default behavior)
• service_healthy: Wait for health check to pass
• service_completed_successfully: Wait for service to exit with status 0

The api service waits for the database to be healthy but only waits for the cache container to start, reflecting that Redis typically becomes ready almost immediately.

Healthcheck Command Formats

Docker supports multiple formats for health check commands:

Shell form:

healthcheck:
  test: pg_isready -U postgres

Exec form (recommended):

healthcheck:
  test: ["CMD", "pg_isready", "-U", "postgres"]

Shell form with shell:

healthcheck:
  test: ["CMD-SHELL", "pg_isready -U postgres || exit 1"]

The exec form (CMD) is preferred because it doesn't invoke a shell, reducing overhead and avoiding shell-specific issues. Use CMD-SHELL when you need shell features like pipes or environment variable expansion.

HTTP-Based Healthchecks

Web services typically implement HTTP health endpoints:

version: '3.8'

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 40s

If curl isn't available in your image, use wget:

healthcheck:
  test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:3000/health"]

Or install a minimal HTTP client during image build and use it for health checks.

Database-Specific Healthchecks

Different databases require different health check approaches:

PostgreSQL:

database:
  image: postgres:14
  healthcheck:
    test: ["CMD", "pg_isready", "-U", "postgres", "-d", "myapp"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 30s

MySQL:

database:
  image: mysql:8
  healthcheck:
    test: ["CMD", "mysqladmin", "ping", "-h", "localhost", "-u", "root", "-p${MYSQL_ROOT_PASSWORD}"]
    interval: 10s
    timeout: 5s
    retries: 3
    start_period: 30s

MongoDB:

database:
  image: mongo:6
  healthcheck:
    test: ["CMD", "mongosh", "--eval", "db.adminCommand('ping')"]
    interval: 10s
    timeout: 5s
    retries: 3
    start_period: 40s

Redis:

cache:
  image: redis:alpine
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 5s
    timeout: 3s
    retries: 3
    start_period: 10s

Advanced Start Period Configuration

The start_period parameter is critical for services with long initialization times:

version: '3.8'

services:
  elasticsearch:
    image: elasticsearch:8.11.0
    environment:
      - discovery.type=single-node
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9200/_cluster/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 120s  # Elasticsearch needs substantial startup time

During the start period:

• Failed health checks don't count toward the retry limit
• The container remains in starting state
• Once start period expires, normal health check rules apply

This prevents premature unhealthy status for services that need significant initialization time.

Composite Healthchecks

Some services need multiple conditions to be truly healthy:

version: '3.8'

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/health && curl -f http://localhost:3000/ready"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s

This checks both a general health endpoint and a readiness endpoint. The health check only passes when both endpoints respond successfully.

Script-Based Healthchecks

Complex health verification logic warrants dedicated health check scripts:

version: '3.8'

services:
  app:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "/app/healthcheck.sh"]
      interval: 20s
      timeout: 10s
      retries: 3
      start_period: 45s

The healthcheck.sh script might contain:

#!/bin/sh
set -e

# Check if application port is listening
nc -z localhost 8080 || exit 1

# Check if database connection works
curl -f http://localhost:8080/api/db/ping || exit 1

# Check if cache is accessible
curl -f http://localhost:8080/api/cache/ping || exit 1

# All checks passed
exit 0

Script-based health checks enable sophisticated verification including multiple service checks, response time validation, and custom application logic.

Disabling Healthchecks

Disable inherited healthchecks when needed:

version: '3.8'

services:
  database:
    image: postgres:14
    healthcheck:
      disable: true

This is useful when:

• The base image defines a healthcheck you don't want
• You're troubleshooting and need to bypass health checking temporarily
• The service doesn't require health monitoring

Dependency Chains

Complex applications often have deep dependency hierarchies:

version: '3.8'

services:
  frontend:
    image: nginx:alpine
    depends_on:
      gateway:
        condition: service_healthy
  
  gateway:
    image: api-gateway:latest
    depends_on:
      auth-service:
        condition: service_healthy
      user-service:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  auth-service:
    image: auth:latest
    depends_on:
      database:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:4000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 25s
  
  user-service:
    image: users:latest
    depends_on:
      database:
        condition: service_healthy
      cache:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 25s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
  
  cache:
    image: redis:alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 10s

This creates a startup sequence:

1. database and cache start first (no dependencies)
2. auth-service and user-service wait for their dependencies
3. gateway waits for both services
4. frontend waits for the gateway

Each service becomes available only after its dependencies are healthy, ensuring proper initialization order.

Parallel Dependency Patterns

Services can depend on multiple services that start in parallel:

version: '3.8'

services:
  aggregator:
    image: aggregator:latest
    depends_on:
      service-a:
        condition: service_healthy
      service-b:
        condition: service_healthy
      service-c:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:6000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s
  
  service-a:
    image: service:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s
  
  service-b:
    image: service:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7001/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s
  
  service-c:
    image: service:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:7002/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s

Services A, B, and C start simultaneously. The aggregator waits for all three to be healthy before starting.

Optional Dependencies

Some dependencies should be waited for if present but not block startup if absent:

version: '3.8'

services:
  app:
    image: myapp:latest
    depends_on:
      database:
        condition: service_healthy
      cache:
        condition: service_started
        required: false
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  cache:
    image: redis:alpine

The app always waits for the database to be healthy but can start even if the cache service doesn't exist or fails to start.

Graceful Degradation with Dependencies

Implement fallback behaviors when optional services are unavailable:

version: '3.8'

services:
  api:
    image: node:18-alpine
    environment:
      - CACHE_ENABLED=true
      - CACHE_HOST=cache
      - FALLBACK_MODE=graceful
    depends_on:
      database:
        condition: service_healthy
      cache:
        condition: service_started
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/health || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  cache:
    image: redis:alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 3s
      retries: 3
      start_period: 10s

The API service can detect cache unavailability and operate with degraded performance rather than failing completely.

Startup Probes vs Liveness Probes

While Docker Compose uses a single healthcheck configuration, you can implement probe-like patterns:

Startup-focused healthcheck:

services:
  slow-starter:
    image: java-app:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/startup"]
      interval: 5s
      timeout: 3s
      retries: 30
      start_period: 120s

Long retry count and start period accommodate slow initialization, switching to failure mode only after extensive attempts.

Liveness-focused healthcheck:

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 20s

Shorter intervals and fewer retries detect deadlocks or hung processes quickly during normal operation.

TCP Socket Healthchecks

For services without HTTP endpoints, use TCP socket checks:

version: '3.8'

services:
  tcp-service:
    image: custom-tcp:latest
    healthcheck:
      test: ["CMD-SHELL", "nc -z localhost 9000 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s

Alternative approaches using different tools:

# Using telnet
healthcheck:
  test: ["CMD-SHELL", "echo -e 'QUIT' | telnet localhost 9000 || exit 1"]

# Using timeout with /dev/tcp (bash only)
healthcheck:
  test: ["CMD-SHELL", "timeout 1 bash -c 'cat < /dev/null > /dev/tcp/localhost/9000' || exit 1"]

Multi-Port Healthchecks

Services exposing multiple ports need comprehensive health verification:

version: '3.8'

services:
  multi-port-service:
    image: myapp:latest
    healthcheck:
      test: ["CMD-SHELL", "nc -z localhost 8080 && nc -z localhost 8443 && nc -z localhost 9090"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s

This ensures HTTP (8080), HTTPS (8443), and metrics (9090) endpoints are all responsive.

Application-Specific Health Logic

Implement domain-specific health checks:

version: '3.8'

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/health | grep -q '\"status\":\"ok\"' || exit 1"]
      interval: 15s
      timeout: 10s
      retries: 3
      start_period: 30s

This validates not just that the endpoint responds but that it returns the expected content, catching cases where the service is running but not functioning correctly.

Database Connection Pool Healthchecks

Verify database connectivity at the application level:

version: '3.8'

services:
  api:
    image: node:18-alpine
    depends_on:
      database:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "node", "/app/health-check.js"]
      interval: 20s
      timeout: 15s
      retries: 3
      start_period: 45s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

The health-check.js script attempts an actual database query, ensuring not just that PostgreSQL is running but that the application can establish connections and execute queries.

Dependency Restart Behavior

When a dependency fails its healthcheck, dependent services don't automatically restart. Configure restart policies to handle dependency failures:

version: '3.8'

services:
  api:
    image: node:18-alpine
    restart: on-failure:5
    depends_on:
      database:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  database:
    image: postgres:14
    restart: unless-stopped
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

If the database becomes unhealthy and restarts, the API's health check will likely fail (unable to reach database), triggering the API's restart policy.

External Service Dependencies

Applications often depend on external services outside the Compose environment:

version: '3.8'

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:3000/health && curl -f https://external-api.example.com/health"]
      interval: 30s
      timeout: 20s
      retries: 5
      start_period: 40s

This health check verifies both internal service health and connectivity to external dependencies, marking the service unhealthy if external services become unreachable.

Cascading Failure Prevention

Design health checks to prevent cascading failures:

version: '3.8'

services:
  gateway:
    image: api-gateway:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 30s
  
  service-a:
    image: service:latest
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9000/health --max-time 3 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s
  
  service-b:
    image: service:latest
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:9001/health --max-time 3 || exit 1"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 20s

Using --max-time in health checks prevents slow-responding services from causing health check timeouts that could trigger false negatives in healthy services.

Monitoring Healthcheck Results

Docker provides commands to inspect health status:

# View health status in docker ps
docker-compose ps

# Get detailed health check logs
docker inspect --format='{{json .State.Health}}' container_name

# Filter for unhealthy containers
docker ps --filter health=unhealthy

Understanding health check history helps diagnose intermittent issues:

docker inspect container_name | jq '.[0].State.Health.Log'

This shows the last several health check results including timestamps and output, revealing patterns in health check failures.

Custom Healthcheck Intervals by Service Type

Different services warrant different health check frequencies:

version: '3.8'

services:
  # Critical, fast-changing service - frequent checks
  cache:
    image: redis:alpine
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 5s
      timeout: 2s
      retries: 3
      start_period: 5s
  
  # Standard API - moderate checks
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 15s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  # Stable database - infrequent checks
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 45s
  
  # Background worker - very infrequent checks
  worker:
    image: python:3.9
    healthcheck:
      test: ["CMD", "python", "/app/health.py"]
      interval: 60s
      timeout: 15s
      retries: 3
      start_period: 20s

This approach balances health monitoring needs against health check overhead.

Healthcheck Exit Codes

Health check scripts should use standard exit codes:

• 0: Healthy
• 1: Unhealthy
• Other non-zero values: Unhealthy

Example health check script with proper exit codes:

version: '3.8'

services:
  api:
    image: node:18-alpine
    healthcheck:
      test: ["CMD", "/app/comprehensive-health.sh"]
      interval: 20s
      timeout: 15s
      retries: 3
      start_period: 40s

The comprehensive-health.sh script:

#!/bin/sh

# Check HTTP endpoint
if ! curl -f http://localhost:3000/health >/dev/null 2>&1; then
    echo "HTTP health endpoint failed"
    exit 1
fi

# Check database connectivity
if ! curl -f http://localhost:3000/db/ping >/dev/null 2>&1; then
    echo "Database connectivity failed"
    exit 1
fi

# Check cache connectivity
if ! curl -f http://localhost:3000/cache/ping >/dev/null 2>&1; then
    echo "Cache connectivity failed"  
    exit 1
fi

# All checks passed
echo "All health checks passed"
exit 0

Dependency Wait Strategies

Implement sophisticated wait strategies for services with complex initialization:

version: '3.8'

services:
  app:
    image: myapp:latest
    depends_on:
      database:
        condition: service_healthy
    entrypoint: ["/app/wait-and-start.sh"]
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 60s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD-SHELL", "pg_isready && psql -U postgres -c 'SELECT 1 FROM pg_database WHERE datname = $$myapp$$' | grep -q 1"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 40s

The database health check verifies both PostgreSQL readiness and that the specific database exists, ensuring the application finds a fully initialized database.

Coordinated Startup with Init Containers

Simulate init container patterns using condition-based dependencies:

version: '3.8'

services:
  db-migrator:
    image: db-migrate:latest
    depends_on:
      database:
        condition: service_healthy
    restart: on-failure
  
  app:
    image: myapp:latest
    depends_on:
      db-migrator:
        condition: service_completed_successfully
      database:
        condition: service_healthy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s
  
  database:
    image: postgres:14
    healthcheck:
      test: ["CMD", "pg_isready"]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 30s

This ensures database migrations run to completion before the application starts, preventing startup race conditions.

Healthcheck Best Practices

Keep health checks lightweight: Health checks run frequently—avoid expensive operations that could impact service performance.

Use appropriate timeouts: Set timeouts shorter than intervals to prevent overlapping health checks. A common pattern is timeout < interval / 2.

Adjust retries for service characteristics: Fast-recovering services need fewer retries; services with intermittent issues benefit from higher retry counts.

Validate actual functionality: Don't just check if a port is open—verify the service can actually perform its core functions.

Consider start periods carefully: Insufficient start periods cause premature unhealthy status; excessive start periods delay failure detection.

Use specific health endpoints: Implement dedicated /health endpoints that verify critical dependencies without performing actual business logic.

Document health check behavior: Comment health check configurations explaining what they verify and why specific values were chosen.

Test health check scripts independently: Run health check commands manually to verify they work as expected and return appropriate exit codes.

Monitor health check logs: Regular review of health check output reveals patterns in service behavior and potential issues.

Balance coverage with overhead: Comprehensive health checks provide better detection but consume resources—find the right balance for your services.

Docker Compose healthchecks and dependency patterns provide the foundation for reliable multi-container applications. By implementing thorough health verification and carefully orchestrating service dependencies, you ensure services start in the correct order, become available at the right time, and maintain operational health throughout their lifecycle.