Container health monitoring extends beyond simply checking if a container is running. A container process may be active while the application inside is unresponsive, stuck, or unable to serve requests. Healthchecks provide a mechanism to verify that containerized applications are not just running, but actually functioning correctly.

Understanding Healthchecks

A healthcheck is a command that runs periodically inside a container to verify the application's health. The command returns an exit code indicating whether the application is healthy. Docker uses this information to mark containers as healthy or unhealthy, enabling informed decisions about traffic routing, container restarts, and operational status.

Health States

Containers with healthchecks transition through three states:

starting: The initial state when a container first launches. The healthcheck hasn't run enough times to determine health status. This grace period allows applications time to initialize before being evaluated.

healthy: The healthcheck command has succeeded. The application is functioning correctly and ready to serve traffic.

unhealthy: The healthcheck command has failed multiple consecutive times. The application is not functioning properly.

Containers without healthchecks never enter these states—they're either running or stopped, with no health assessment.

Configuring Healthchecks

Healthchecks are defined in your configuration using the healthcheck key under each service.

Basic Healthcheck Structure

The simplest healthcheck configuration includes a test command:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]

This runs curl -f http://localhost:8080/health inside the container. The -f flag makes curl exit with a non-zero code if the HTTP response indicates an error (4xx or 5xx status).

Alternative Command Formats

Healthchecks support multiple command formats:

Array format (recommended):

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]

Shell format:

healthcheck:
  test: curl -f http://localhost:8080/health

The shell format wraps your command in /bin/sh -c, providing shell features like pipes and redirects. The array format executes commands directly without shell interpretation, offering better performance and avoiding shell injection concerns.

CMD vs CMD-SHELL:

# Direct execution
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]

# Shell execution
healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost:8080/health || exit 1"]

CMD-SHELL explicitly indicates shell execution, useful when you need shell features in array format.

Healthcheck Parameters

Beyond the test command, several parameters control healthcheck behavior.

Interval

The interval parameter defines how often the healthcheck runs:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost/health"]
  interval: 30s

This runs the healthcheck every 30 seconds. Default is 30 seconds if not specified.

Shorter intervals detect problems faster but increase system load. Longer intervals reduce overhead but delay problem detection. Choose intervals based on your application's characteristics:

Fast-changing state: 10-15 seconds
Stable services: 30-60 seconds
Low-priority background workers: 60-120 seconds

Timeout

The timeout parameter limits how long a healthcheck can run:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost/health"]
  interval: 30s
  timeout: 10s

If the healthcheck doesn't complete within 10 seconds, it's considered failed. Default timeout is 30 seconds.

Set timeouts shorter than intervals to prevent overlapping healthchecks. If your application responds in milliseconds under normal conditions, set a generous timeout (3-5 seconds) to allow for occasional slowdowns without false failures.

Retries

The retries parameter determines how many consecutive failures trigger an unhealthy state:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost/health"]
  interval: 30s
  timeout: 10s
  retries: 3

The container becomes unhealthy only after 3 consecutive failures. This prevents transient glitches from incorrectly marking healthy containers as unhealthy.

Default is 3 retries. Increase for services that occasionally have temporary issues. Decrease for critical services where you want fast failure detection.

Start Period

The start_period gives applications time to initialize:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 60s

For the first 60 seconds after container start, failed healthchecks don't count toward the retry limit. This grace period prevents applications with long initialization times from being marked unhealthy during startup.

Failed healthchecks during the start period still extend the starting state—the container doesn't become healthy until a healthcheck succeeds. But failures don't count against the retry limit.

The start period begins when the container starts, not when the first healthcheck runs. Set it longer than your application's maximum initialization time.

Complete Example

A fully configured healthcheck:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 40s

This configuration:

Checks health every 30 seconds
Fails the check if it takes longer than 10 seconds
Requires 3 consecutive failures to mark unhealthy
Allows 40 seconds for application initialization

Healthcheck Commands

The effectiveness of healthchecks depends on choosing appropriate test commands.

HTTP Endpoint Checks

For web applications, HTTP endpoint checks are most common:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]

This assumes your application exposes a /health endpoint that returns HTTP 200 when healthy.

Using wget instead of curl:

healthcheck:
  test: ["CMD", "wget", "--quiet", "--tries=1", "--spider", "http://localhost:8080/health"]

The --spider flag makes wget check if the URL exists without downloading content. --quiet suppresses output, and --tries=1 prevents retries.

TCP Port Checks

Test if a service is listening on a port:

healthcheck:
  test: ["CMD-SHELL", "nc -z localhost 5432 || exit 1"]

This uses netcat (nc) to check if port 5432 accepts connections. The -z flag performs a connection test without sending data.

For services that must accept connections but don't have HTTP endpoints, TCP checks verify the service is listening.

Process Checks

Verify a specific process is running:

healthcheck:
  test: ["CMD-SHELL", "pgrep -f 'python app.py' || exit 1"]

This checks if a process matching python app.py exists. If found, the healthcheck succeeds.

Process checks are rudimentary—they confirm the process exists but not that it's functioning correctly. Use them only when better alternatives aren't available.

Command Execution Checks

Test application functionality directly:

healthcheck:
  test: ["CMD", "python", "/app/healthcheck.py"]

This executes a custom script that performs application-specific health verification. The script exits with 0 for healthy, non-zero for unhealthy.

Custom healthcheck scripts can:

Query databases and verify connectivity
Check cache availability
Verify file system access
Test external service connections
Validate application state

Database Checks

For database containers:

PostgreSQL:

healthcheck:
  test: ["CMD", "pg_isready", "-U", "postgres"]

MySQL:

healthcheck:
  test: ["CMD", "mysqladmin", "ping", "-h", "localhost"]

MongoDB:

healthcheck:
  test: ["CMD", "mongo", "--eval", "db.adminCommand('ping')"]

Redis:

healthcheck:
  test: ["CMD", "redis-cli", "ping"]

These commands use database-specific tools to verify the database accepts connections and responds to queries.

Viewing Health Status

Monitor container health through several mechanisms.

Checking Current Status

View health status of all containers:

docker compose ps

The output includes a "Status" column showing health state. Healthy containers show "(healthy)", unhealthy ones show "(unhealthy)".

Detailed Health Information

Inspect detailed health information:

docker inspect <container-name>

This shows the health section with:

Current health status
Failed healthcheck count
Last healthcheck output
Healthcheck execution history

The health history includes timestamps, exit codes, and output from recent healthchecks, valuable for debugging why a container is unhealthy.

Monitoring Health Changes

Watch for health status changes:

docker events --filter event=health_status

This streams health status change events as they occur. You'll see when containers transition between healthy and unhealthy states in real-time.

Healthcheck Best Practices

Effective healthchecks follow specific patterns and principles.

Keep Checks Lightweight

Healthchecks run frequently and shouldn't consume significant resources. A healthcheck that takes seconds to execute or uses substantial CPU/memory degrades overall system performance.

Good healthchecks:

Complete in milliseconds
Use minimal CPU and memory
Don't write to disk unnecessarily
Avoid complex computations

Bad healthchecks:

Perform expensive database queries
Generate large reports
Execute heavy computations
Test every possible code path

Test Actual Functionality

Don't just check if the process is running—verify it can actually serve requests.

Poor healthcheck:

healthcheck:
  test: ["CMD", "ps", "aux"]

This only confirms processes exist, not that they work.

Better healthcheck:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]

This verifies the web server accepts and responds to HTTP requests.

Create Dedicated Health Endpoints

Applications should expose endpoints specifically for health checking:

GET /health -> 200 OK if healthy, 503 Service Unavailable if unhealthy

Health endpoints should:

Return quickly (under 100ms)
Check critical dependencies
Return appropriate HTTP status codes
Include minimal response bodies

The endpoint might verify:

Database connectivity
Cache accessibility
Critical file system paths
Memory availability

But it should avoid slow operations like:

Full database scans
Complex queries
External API calls with long timeouts
File system scans

Handle Dependencies Appropriately

Consider whether healthchecks should verify dependency availability.

Include dependency checks when:

The application cannot function without the dependency
Requests will fail if the dependency is unavailable
You want the container marked unhealthy when dependencies fail

Exclude dependency checks when:

Dependency failures are temporary and recoverable
The application can operate in degraded mode
You want to separate application health from dependency health

For example, a web API might check database connectivity in its healthcheck because it cannot serve requests without the database. But it might not check an external payment API because payment failures should be handled gracefully without marking the entire service unhealthy.

Set Realistic Timeouts

Timeouts should accommodate normal operation plus a margin:

If your health endpoint typically responds in 50ms, set timeout to 3-5 seconds. This allows for occasional slowdowns without false failures.

Too-short timeouts cause false failures during temporary slowdowns. Too-long timeouts delay detection of real problems.

Use Appropriate Intervals

Balance between detection speed and system load:

Critical user-facing services: 10-15 seconds
Background workers: 30-60 seconds
Infrastructure services: 20-30 seconds

More frequent checks detect problems faster but consume more resources.

Configure Adequate Start Periods

Applications with long initialization need generous start periods:

Database containers: 30-60 seconds
Applications with large dependency trees: 60-120 seconds
Services loading large datasets: 120+ seconds

Set start periods to at least 1.5x your application's typical initialization time.

Troubleshooting Healthchecks

When healthchecks don't behave as expected, systematic troubleshooting identifies the issue.

Container Stuck in Starting

If a container remains in "starting" state indefinitely:

Check if healthchecks are passing at all
Verify the healthcheck command syntax
Ensure required tools (curl, wget) are installed in the container
Test the healthcheck command manually inside the container
Check application logs for initialization errors

Manually execute the healthcheck:

docker exec <container-name> curl -f http://localhost:8080/health

If this fails, the healthcheck is correctly detecting a problem. Fix the application issue.

Healthchecks Failing Immediately

When healthchecks fail right after container start:

Verify the application is actually listening on the checked port
Ensure the health endpoint exists and returns appropriate status codes
Check if the start period is long enough for initialization
Confirm network connectivity inside the container

Intermittent Failures

For healthchecks that occasionally fail:

Increase the timeout to allow for temporary slowdowns
Increase retries to tolerate transient issues
Check if healthchecks coincide with resource-intensive operations
Verify the health endpoint isn't rate-limited

Healthcheck Command Not Found

If healthchecks fail with "command not found":

The container image doesn't include the required tool (curl, wget, nc). Either:

Use a different tool that's available
Build a custom image that includes the tool
Write a healthcheck script using tools available in the base image

Most minimal images don't include curl or wget by default. Alpine-based images need explicit installation of these tools.

False Negatives (Healthy When Unhealthy)

If containers are marked healthy despite being broken:

The healthcheck isn't testing actual functionality
The health endpoint always returns success
The healthcheck command has incorrect logic

Improve the healthcheck to verify real application behavior.

False Positives (Unhealthy When Healthy)

If healthy containers are marked unhealthy:

Timeout is too short
Retries are too few
Start period is insufficient
Health endpoint is too slow
Healthcheck runs too frequently during load spikes

Adjust parameters to be more forgiving or optimize the health endpoint.

Advanced Healthcheck Patterns

Gradual Health Degradation

Some applications degrade gradually rather than failing completely. Implement tiered health responses:

/health/liveness: Returns 200 if the process is alive (minimal check)
/health/readiness: Returns 200 if ready to serve traffic (includes dependencies)

Use liveness checks for healthcheck configuration. Use readiness checks in load balancers to control traffic routing.

Dependency Chain Verification

For complex service dependencies, verify the entire chain:

healthcheck:
  test: ["CMD", "python", "/app/check_dependencies.py"]

The script verifies:

Database connectivity
Cache accessibility
Message queue connection
Critical file paths exist

Return success only if all dependencies are available.

Self-Healing Applications

Combine healthchecks with restart policies for self-healing:

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 10s
  retries: 3
restart: on-failure

When healthchecks fail repeatedly, the restart policy automatically restarts the container, potentially recovering from transient issues.

Conditional Health Logic

Health endpoints can implement conditional logic:

@app.route('/health')
def health():
    if not database_available():
        return ('Database unavailable', 503)
    
    if cache_connection_count() > MAX_CONNECTIONS:
        return ('Connection pool exhausted', 503)
    
    if free_memory() < MIN_MEMORY:
        return ('Low memory', 503)
    
    return ('OK', 200)

This checks multiple conditions and returns unhealthy if any fail.

Healthchecks and Container Lifecycle

Understanding how healthchecks interact with container lifecycle helps predict behavior.

During Startup

When a container starts:

Container enters "starting" state
Healthchecks begin after the container is running
Failed healthchecks during start period don't count against retries
First successful healthcheck transitions to "healthy"
If healthchecks never succeed, container remains "starting" indefinitely

During Normal Operation

While running:

Healthchecks run at configured intervals
Successful checks maintain "healthy" state
Failed checks increment failure counter
After consecutive failures exceed retries, state becomes "unhealthy"
A single success resets failure counter and returns to "healthy"

During Shutdown

When stopping containers:

Stop signal is sent immediately
Healthchecks may still run during graceful shutdown period
Health state becomes irrelevant once container stops
Healthcheck results don't affect shutdown process

With Restart Policies

Restart policies interact with health status:

restart: always: Restarts regardless of health status
restart: on-failure: Restarts unhealthy containers
restart: unless-stopped: Continues restarting unhealthy containers

Unhealthy status doesn't automatically trigger restarts unless a restart policy is configured.

Disabling Healthchecks

Inherited healthchecks (from base images) can be disabled:

healthcheck:
  disable: true

This completely removes healthcheck functionality for the service. Use this when:

The inherited healthcheck is incorrect
You want to test without health monitoring
The healthcheck causes problems

Multiple Service Healthchecks

Configure healthchecks independently for each service:

web:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
    interval: 15s
    timeout: 5s
    retries: 3
    start_period: 30s

database:
  healthcheck:
    test: ["CMD", "pg_isready", "-U", "postgres"]
    interval: 30s
    timeout: 10s
    retries: 5
    start_period: 60s

cache:
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 20s
    timeout: 3s
    retries: 3
    start_period: 10s

Each service has tailored healthcheck parameters matching its characteristics.

Healthcheck Output

Healthcheck commands can produce output that aids debugging:

healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost:8080/health || echo 'Health check failed'"]

Output from healthchecks appears in the container's health information (viewable with docker inspect). Keep output minimal—verbose healthcheck output pollutes logs.

Common Healthcheck Patterns by Service Type

Web Applications

healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
  interval: 30s
  timeout: 5s
  retries: 3
  start_period: 40s

Databases

healthcheck:
  test: ["CMD", "pg_isready", "-U", "postgres"]
  interval: 30s
  timeout: 10s
  retries: 5
  start_period: 60s

Message Queues

healthcheck:
  test: ["CMD", "rabbitmqctl", "status"]
  interval: 30s
  timeout: 10s
  retries: 3
  start_period: 60s

Caches

healthcheck:
  test: ["CMD", "redis-cli", "ping"]
  interval: 20s
  timeout: 3s
  retries: 3
  start_period: 10s

Background Workers

healthcheck:
  test: ["CMD", "python", "/app/worker_health.py"]
  interval: 60s
  timeout: 10s
  retries: 3
  start_period: 30s

Real-World Examples

E-commerce Application

api:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:3000/api/health"]
    interval: 20s
    timeout: 5s
    retries: 3
    start_period: 45s

database:
  healthcheck:
    test: ["CMD", "pg_isready", "-U", "ecommerce"]
    interval: 30s
    timeout: 10s
    retries: 5
    start_period: 60s

redis:
  healthcheck:
    test: ["CMD", "redis-cli", "ping"]
    interval: 15s
    timeout: 3s
    retries: 3
    start_period: 20s

Microservices Architecture

user-service:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
    interval: 15s
    timeout: 5s
    retries: 3
    start_period: 30s

order-service:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8081/health"]
    interval: 15s
    timeout: 5s
    retries: 3
    start_period: 30s

notification-service:
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:8082/health"]
    interval: 20s
    timeout: 5s
    retries: 3
    start_period: 40s

Conclusion

Healthchecks transform container monitoring from binary "running/stopped" states to meaningful "healthy/unhealthy" assessments. Properly configured healthchecks detect application failures, enable informed operational decisions, and form the foundation for reliable containerized deployments. Invest time in crafting appropriate healthchecks for each service, tune parameters based on application behavior, and regularly test healthcheck effectiveness to maintain system reliability.

Docker Compose Healthchecks: Complete Guide to Container Health Monitoring