Back to BlogDocker compose · docker · multi-environment

Managing Multi-Environment Architecture with Docker Compose

2025-12-24

Modern application development requires running the same services across multiple environments: development, staging, production, and testing. Each environment has different requirements for configuration, resource allocation, and service dependencies. Docker Compose provides powerful mechanisms to manage these variations through override files and profiles, allowing you to maintain a single source of truth while accommodating environment-specific needs.

Understanding Base and Override File Structure

Docker Compose follows a layered configuration approach. The foundation is your base docker-compose.yml file, which contains common service definitions shared across all environments. This base file defines the essential structure of your application stack.

When you run Docker Compose, it automatically looks for and merges multiple files in a specific order:

  1. docker-compose.yml (base file)
  2. docker-compose.override.yml (default override)

Additional override files can be specified explicitly using the -f flag. Files are merged in the order they're specified, with later files taking precedence over earlier ones.

Creating a Base Configuration

Your base docker-compose.yml should contain service definitions that are common to all environments. This includes service names, image specifications, and configuration that doesn't change between environments.

version: '3.8'

services:
  web:
    image: nginx:alpine
    ports:
      - "8080:80"
    
  api:
    build:
      context: ./api
    environment:
      - NODE_ENV=production
    
  database:
    image: postgres:14
    environment:
      - POSTGRES_DB=myapp

This base file establishes the core architecture. All environments will have these three services, but specific configurations will vary.

Using docker-compose.override.yml for Local Development

The docker-compose.override.yml file is automatically applied when you run docker-compose up without additional flags. This is ideal for local development configurations.

version: '3.8'

services:
  web:
    volumes:
      - ./nginx/conf:/etc/nginx/conf.d
    ports:
      - "80:80"
    
  api:
    build:
      context: ./api
      target: development
    volumes:
      - ./api:/usr/src/app
      - /usr/src/app/node_modules
    environment:
      - NODE_ENV=development
      - DEBUG=true
    command: npm run dev
    
  database:
    ports:
      - "5432:5432"
    environment:
      - POSTGRES_PASSWORD=devpassword

This override adds development-specific features like volume mounts for live code reloading, exposed database ports for direct access, and debug flags. When you run docker-compose up, both files merge automatically.

Creating Environment-Specific Override Files

For production, staging, or testing environments, create separate override files with descriptive names:

docker-compose.prod.yml:

version: '3.8'

services:
  web:
    restart: always
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
    
  api:
    image: myregistry.com/api:latest
    restart: always
    environment:
      - NODE_ENV=production
      - API_WORKERS=4
    
  database:
    restart: always
    environment:
      - POSTGRES_PASSWORD=${DB_PASSWORD}
    logging:
      driver: "json-file"
      options:
        max-size: "50m"
        max-file: "5"

docker-compose.staging.yml:

version: '3.8'

services:
  web:
    environment:
      - ENVIRONMENT=staging
    
  api:
    image: myregistry.com/api:staging
    environment:
      - NODE_ENV=staging
      - DEBUG=true
      - API_WORKERS=2
    
  database:
    environment:
      - POSTGRES_PASSWORD=${STAGING_DB_PASSWORD}

To use these files, specify them explicitly:

docker-compose -f docker-compose.yml -f docker-compose.prod.yml up -d

Understanding File Merge Behavior

When Docker Compose merges files, it follows specific rules:

For scalar values (strings, numbers, booleans), the later file completely replaces the earlier value:

# Base file
services:
  api:
    environment:
      - NODE_ENV=production

# Override file
services:
  api:
    environment:
      - NODE_ENV=development
      - DEBUG=true

# Result: NODE_ENV=development, DEBUG=true

For lists and arrays, items are concatenated:

# Base file
services:
  web:
    ports:
      - "8080:80"

# Override file
services:
  web:
    ports:
      - "443:443"

# Result: Both port mappings are active

For mappings (key-value pairs), keys are merged, with later values overriding earlier ones:

# Base file
services:
  api:
    environment:
      NODE_ENV: production
      LOG_LEVEL: info

# Override file
services:
  api:
    environment:
      NODE_ENV: development
      DEBUG: "true"

# Result: NODE_ENV=development, LOG_LEVEL=info, DEBUG=true

Working with Profiles

Profiles provide an alternative approach to managing environments by allowing you to selectively enable or disable services. This is particularly useful when different environments require different service combinations.

Define profiles in your compose file:

version: '3.8'

services:
  web:
    image: nginx:alpine
    # No profile - always runs
    
  api:
    build: ./api
    # No profile - always runs
    
  database:
    image: postgres:14
    # No profile - always runs
    
  redis:
    image: redis:alpine
    profiles:
      - caching
    
  monitoring:
    image: prom/prometheus
    profiles:
      - monitoring
      
  testing:
    build: ./tests
    profiles:
      - test

By default, only services without profiles run. To activate profile-specific services:

# Run with caching enabled
docker-compose --profile caching up

# Run with multiple profiles
docker-compose --profile caching --profile monitoring up

# Run only test profile
docker-compose --profile test up

Combining Profiles with Override Files

Profiles and override files work together seamlessly. You can define profiles in your base file and override their configurations in environment-specific files:

docker-compose.yml:

version: '3.8'

services:
  api:
    build: ./api
    
  cache:
    image: redis:alpine
    profiles:
      - caching

docker-compose.dev.yml:

version: '3.8'

services:
  cache:
    ports:
      - "6379:6379"
    command: redis-server --loglevel verbose

docker-compose.prod.yml:

version: '3.8'

services:
  cache:
    command: redis-server --maxmemory 256mb --maxmemory-policy allkeys-lru
    restart: always

Usage:

# Development with caching
docker-compose -f docker-compose.yml -f docker-compose.dev.yml --profile caching up

# Production with caching
docker-compose -f docker-compose.yml -f docker-compose.prod.yml --profile caching up

Using Environment Variables in Override Files

Environment variables provide dynamic configuration across different deployments of the same environment. You can use .env files and variable substitution:

.env:

DB_PASSWORD=secretpassword
API_PORT=3000
IMAGE_TAG=v1.2.3

docker-compose.yml:

version: '3.8'

services:
  api:
    image: myapp/api:${IMAGE_TAG:-latest}
    ports:
      - "${API_PORT:-3000}:3000"
    
  database:
    image: postgres:14
    environment:
      - POSTGRES_PASSWORD=${DB_PASSWORD}

You can create different .env files for each environment:

  • .env.development
  • .env.staging
  • .env.production

Load the appropriate file:

docker-compose --env-file .env.production up

Practical Multi-Environment Workflow

Here's a complete workflow structure for managing multiple environments:

File structure:

project/
├── docker-compose.yml          # Base configuration
├── docker-compose.override.yml # Local development (auto-loaded)
├── docker-compose.prod.yml     # Production overrides
├── docker-compose.staging.yml  # Staging overrides
├── docker-compose.test.yml     # Testing overrides
├── .env.development
├── .env.staging
└── .env.production

docker-compose.yml (Base):

version: '3.8'

services:
  web:
    image: nginx:alpine
    
  api:
    build:
      context: ./api
    depends_on:
      - database
    
  database:
    image: postgres:14
    
  worker:
    build:
      context: ./worker
    profiles:
      - background-jobs

docker-compose.override.yml (Development):

version: '3.8'

services:
  web:
    volumes:
      - ./web:/usr/share/nginx/html
    ports:
      - "80:80"
    
  api:
    volumes:
      - ./api:/app
    environment:
      - HOT_RELOAD=true
    
  database:
    ports:
      - "5432:5432"
    
  worker:
    volumes:
      - ./worker:/app
    environment:
      - DEV_MODE=true

docker-compose.staging.yml:

version: '3.8'

services:
  web:
    image: myregistry.com/web:${IMAGE_TAG}
    
  api:
    image: myregistry.com/api:${IMAGE_TAG}
    environment:
      - ENVIRONMENT=staging
    
  worker:
    image: myregistry.com/worker:${IMAGE_TAG}

docker-compose.prod.yml:

version: '3.8'

services:
  web:
    image: myregistry.com/web:${IMAGE_TAG}
    restart: always
    
  api:
    image: myregistry.com/api:${IMAGE_TAG}
    restart: always
    environment:
      - ENVIRONMENT=production
    
  database:
    restart: always
    
  worker:
    image: myregistry.com/worker:${IMAGE_TAG}
    restart: always

Commands for each environment:

# Local development (automatic override)
docker-compose up

# Local with background jobs
docker-compose --profile background-jobs up

# Staging
docker-compose -f docker-compose.yml -f docker-compose.staging.yml --env-file .env.staging up -d

# Production
docker-compose -f docker-compose.yml -f docker-compose.prod.yml --env-file .env.production up -d

# Production with background jobs
docker-compose -f docker-compose.yml -f docker-compose.prod.yml --env-file .env.production --profile background-jobs up -d

Advanced Profile Patterns

Profiles can be used creatively to manage complex service dependencies and optional components.

Feature flags with profiles:

version: '3.8'

services:
  api:
    build: ./api
    
  database:
    image: postgres:14
    
  search:
    image: elasticsearch:8
    profiles:
      - search-enabled
    
  analytics:
    image: clickhouse/clickhouse-server
    profiles:
      - analytics-enabled
    
  payment-processor:
    build: ./payment
    profiles:
      - payments
      - full
    
  notification-service:
    build: ./notifications
    profiles:
      - notifications
      - full

This structure allows you to:

# Minimal setup
docker-compose up

# With search functionality
docker-compose --profile search-enabled up

# Full production setup
docker-compose --profile full up

Environment-based profiles:

version: '3.8'

services:
  api:
    build: ./api
    
  debug-tools:
    image: nicolaka/netshoot
    profiles:
      - debug
    command: sleep infinity
    
  performance-monitor:
    image: grafana/grafana
    profiles:
      - monitoring
    
  load-generator:
    image: williamyeh/wrk
    profiles:
      - load-test

Overriding Specific Service Properties

Sometimes you need to override only specific properties while keeping most of the base configuration intact.

Overriding command:

# Base
services:
  api:
    image: myapp/api
    command: npm start

# Override
services:
  api:
    command: npm run start:debug

Extending environment variables:

# Base
services:
  api:
    environment:
      - NODE_ENV=production
      - PORT=3000

# Override - adds to existing
services:
  api:
    environment:
      - DEBUG=true
      - LOG_LEVEL=verbose

Changing build targets:

# Base
services:
  api:
    build:
      context: ./api

# Override for development
services:
  api:
    build:
      context: ./api
      target: development
      args:
        - NODE_ENV=development

Managing Database Initialization Across Environments

Different environments often need different database initialization strategies:

version: '3.8'

services:
  database:
    image: postgres:14
    environment:
      - POSTGRES_DB=myapp

docker-compose.override.yml (Development):

services:
  database:
    volumes:
      - ./init-scripts/dev:/docker-entrypoint-initdb.d
      - dev-db-data:/var/lib/postgresql/data

volumes:
  dev-db-data:

docker-compose.staging.yml:

services:
  database:
    volumes:
      - ./init-scripts/staging:/docker-entrypoint-initdb.d
      - staging-db-data:/var/lib/postgresql/data

volumes:
  staging-db-data:

docker-compose.test.yml:

services:
  database:
    volumes:
      - ./init-scripts/test:/docker-entrypoint-initdb.d
    # No named volume - uses temporary storage

  test-runner:
    build: ./tests
    depends_on:
      - database
    profiles:
      - test

Service Variants with Profiles

Profiles enable you to run different versions of the same logical service:

version: '3.8'

services:
  api-standard:
    build: ./api
    environment:
      - VARIANT=standard
    # Default - no profile needed
    
  api-premium:
    build:
      context: ./api
      args:
        - FEATURES=premium
    environment:
      - VARIANT=premium
    profiles:
      - premium
    ports:
      - "3001:3000"

Running different configurations:

# Standard API only
docker-compose up

# Both standard and premium
docker-compose --profile premium up

Conditional Service Dependencies

Use profiles to manage optional service dependencies:

version: '3.8'

services:
  api:
    build: ./api
    depends_on:
      - database
    
  database:
    image: postgres:14
    
  cache:
    image: redis:alpine
    profiles:
      - with-cache
    
  api-cached:
    build: ./api
    environment:
      - CACHE_ENABLED=true
      - REDIS_HOST=cache
    depends_on:
      - database
      - cache
    profiles:
      - with-cache
    ports:
      - "3001:3000"

This allows you to run the API with or without caching:

# Without cache
docker-compose up api

# With cache
docker-compose --profile with-cache up api-cached

Testing Environment Configurations

Create specialized test configurations that modify service behavior:

docker-compose.test.yml:

version: '3.8'

services:
  api:
    environment:
      - NODE_ENV=test
      - DATABASE_URL=postgresql://postgres:testpass@database:5432/testdb
    command: npm run test:integration
    
  database:
    environment:
      - POSTGRES_DB=testdb
      - POSTGRES_PASSWORD=testpass
    
  test-reporter:
    build:
      context: ./tests
      dockerfile: Dockerfile.reporter
    depends_on:
      - api
    profiles:
      - test

Run tests:

docker-compose -f docker-compose.yml -f docker-compose.test.yml --profile test up --abort-on-container-exit

Environment-Specific Port Mappings

Different environments often require different port configurations:

Base file:

services:
  api:
    build: ./api
    expose:
      - "3000"

Development override:

services:
  api:
    ports:
      - "3000:3000"  # Direct access for development

Staging override:

services:
  api:
    ports:
      - "8080:3000"  # Non-standard port for staging

Production override:

services:
  api:
    # No ports exposed - accessed through reverse proxy only
    expose:
      - "3000"

Managing Build Contexts Across Environments

Override build configurations for different environments:

Base:

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile

Development:

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.dev
      args:
        - BUILD_ENV=development
    volumes:
      - ./frontend/src:/app/src

Production:

services:
  frontend:
    build:
      context: ./frontend
      dockerfile: Dockerfile.prod
      args:
        - BUILD_ENV=production
        - OPTIMIZE=true

Environment File Precedence

Understanding how environment files are loaded is crucial:

  1. Environment variables set in the shell
  2. Variables from .env file
  3. Variables defined in docker-compose.yml
  4. Variables defined in override files

Example:

# docker-compose.yml
services:
  api:
    environment:
      - LOG_LEVEL=${LOG_LEVEL:-info}

Priority order (highest to lowest):

# 1. Shell variable (highest priority)
export LOG_LEVEL=debug
docker-compose up

# 2. Custom env file
docker-compose --env-file .env.custom up

# 3. Default .env file
# 4. Default value in compose file (lowest priority)

Profile-Based Environment Selection

Combine profiles with environment-specific configurations:

version: '3.8'

services:
  api:
    build: ./api
    
  api-dev:
    build:
      context: ./api
      target: development
    environment:
      - NODE_ENV=development
    volumes:
      - ./api:/app
    profiles:
      - dev
    
  api-prod:
    image: myregistry.com/api:latest
    environment:
      - NODE_ENV=production
    profiles:
      - prod

Usage:

# Development
docker-compose --profile dev up

# Production
docker-compose --profile prod up

Organizing Complex Multi-Environment Projects

For large projects with many services and environments, consider this structure:

project/
├── docker-compose.yml
├── compose/
│   ├── base/
│   │   ├── api.yml
│   │   ├── database.yml
│   │   └── cache.yml
│   ├── dev/
│   │   ├── overrides.yml
│   │   └── debug-tools.yml
│   ├── staging/
│   │   └── overrides.yml
│   └── prod/
│       └── overrides.yml
└── .env.example

docker-compose.yml:

version: '3.8'

include:
  - compose/base/api.yml
  - compose/base/database.yml
  - compose/base/cache.yml

Load environment-specific configurations:

# Development
docker-compose -f docker-compose.yml -f compose/dev/overrides.yml up

# Production
docker-compose -f docker-compose.yml -f compose/prod/overrides.yml up

Dynamic Service Scaling with Profiles

Use profiles to define different scaling configurations:

version: '3.8'

services:
  api-single:
    build: ./api
    # Default single instance
    
  api-scaled:
    build: ./api
    profiles:
      - scaled
    environment:
      - INSTANCE_ID=${INSTANCE_ID}

docker-compose.scaled.yml:

version: '3.8'

services:
  api-scaled:
    scale: 3

Run scaled configuration:

docker-compose -f docker-compose.yml -f docker-compose.scaled.yml --profile scaled up

Handling Third-Party Service Variations

Different environments may require different external services:

version: '3.8'

services:
  api:
    build: ./api
    
  database-local:
    image: postgres:14
    environment:
      - POSTGRES_PASSWORD=devpass
    profiles:
      - local-db
    
  database-cloud:
    image: postgres:14
    environment:
      - POSTGRES_HOST=${CLOUD_DB_HOST}
      - POSTGRES_PASSWORD=${CLOUD_DB_PASSWORD}
    profiles:
      - cloud-db

Switch between local and cloud databases:

# Local development
docker-compose --profile local-db up

# Using cloud database
docker-compose --profile cloud-db up

Best Practices for Multi-Environment Management

Keep base files minimal: Include only truly common configurations in your base file. Everything that varies should be in override files.

Use descriptive file names: Name override files clearly: docker-compose.prod.yml, not docker-compose.2.yml.

Document required environment variables: Maintain a .env.example file showing all variables each environment needs.

Test environment transitions: Regularly test that switching between environments works smoothly without manual intervention.

Use profiles for optional features: Reserve profiles for truly optional services or features that may not run in every environment.

Validate merged configurations: Use docker-compose config to see the final merged configuration:

docker-compose -f docker-compose.yml -f docker-compose.prod.yml config

Create environment-specific scripts: Write simple shell scripts to encapsulate common environment switches:

#!/bin/bash
# start-staging.sh
docker-compose -f docker-compose.yml \
  -f docker-compose.staging.yml \
  --env-file .env.staging \
  up -d

Version control all compose files: Keep all compose files in version control, but exclude .env files containing sensitive data.

Use consistent naming conventions: Establish clear patterns for service names, profile names, and file names across your project.

Multi-environment management with Docker Compose provides the flexibility to run identical application stacks with environment-appropriate configurations. By leveraging override files and profiles effectively, you can maintain clean, maintainable infrastructure as code that scales from development laptops to production clusters.

All Posts
We use cookies to improve your experience and analyse site traffic. See our Privacy Policy.