Using Docker Compose with Docker-in-Docker Setups

Docker-in-Docker (DinD) enables running Docker commands and Docker Compose inside containers, creating isolated Docker environments perfect for CI/CD pipelines, testing, and development workflows. Understanding how to properly configure Docker-in-Docker setups, manage Docker socket access, handle networking, and troubleshoot common issues allows you to build sophisticated containerized build systems and testing environments.

Understanding Docker-in-Docker Concepts

Docker-in-Docker runs a complete Docker daemon inside a container, creating a nested Docker environment. There are two primary approaches:

True Docker-in-Docker (DinD): Runs a Docker daemon inside a container Docker socket mounting: Shares the host's Docker daemon with containers

Each approach has distinct use cases, security implications, and operational characteristics.

Basic Docker-in-Docker Setup

The official Docker-in-Docker image enables running Docker inside containers:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - docker-certs:/certs/client
      - docker-data:/var/lib/docker

volumes:
  docker-certs:
  docker-data:

The privileged: true flag is required for DinD to manage its own container runtime.

Running Compose in Docker-in-Docker

Create a service that uses the DinD daemon:

version: '3.8'

services:
  docker:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - docker-certs:/certs
      - docker-data:/var/lib/docker
  
  compose-runner:
    image: docker/compose:latest
    depends_on:
      - docker
    environment:
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
    volumes:
      - docker-certs:/certs:ro
      - ./compose-files:/compose
    working_dir: /compose
    command: docker-compose up

volumes:
  docker-certs:
  docker-data:

The compose-runner service connects to the DinD daemon via TCP.

Docker Socket Mounting Approach

Mount the host Docker socket for simpler access:

version: '3.8'

services:
  compose-runner:
    image: docker/compose:latest
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./compose-files:/compose
    working_dir: /compose
    command: docker-compose up

This approach shares the host's Docker daemon, making it simpler but with different security implications.

CI/CD Pipeline with DinD

Create a complete CI/CD environment:

version: '3.8'

services:
  docker-daemon:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - docker-certs:/certs
      - docker-data:/var/lib/docker
    command: ["dockerd", "--host=tcp://0.0.0.0:2376", "--host=unix:///var/run/docker.sock"]
  
  ci-runner:
    image: docker:24
    depends_on:
      - docker-daemon
    environment:
      - DOCKER_HOST=tcp://docker-daemon:2376
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
    volumes:
      - docker-certs:/certs:ro
      - ./project:/workspace
    working_dir: /workspace
    command: sh -c "
      docker-compose build &&
      docker-compose up -d &&
      docker-compose exec -T app npm test &&
      docker-compose down
      "

volumes:
  docker-certs:
  docker-data:

This setup builds, tests, and tears down services entirely within containers.

Handling TLS Certificates

Configure TLS for secure DinD communication:

version: '3.8'

services:
  docker:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
      - DOCKER_TLS_VERIFY=1
    volumes:
      - docker-certs-ca:/certs/ca
      - docker-certs-client:/certs/client
      - docker-data:/var/lib/docker
  
  client:
    image: docker:24
    depends_on:
      - docker
    environment:
      - DOCKER_HOST=tcp://docker:2376
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
    volumes:
      - docker-certs-client:/certs/client:ro
    command: docker info

volumes:
  docker-certs-ca:
  docker-certs-client:
  docker-data:

TLS ensures encrypted communication between clients and the DinD daemon.

Multi-Stage Build Environment

Use DinD for isolated multi-stage builds:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
  
  builder:
    image: docker:24
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: sh -c "
      docker build -t myapp:builder --target builder . &&
      docker build -t myapp:production --target production . &&
      docker save myapp:production -o /workspace/app.tar
      "

volumes:
  docker-data:

This builds images in isolated environments and exports them.

Compose with BuildKit in DinD

Enable BuildKit for faster builds:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
      - DOCKER_BUILDKIT=1
    volumes:
      - docker-data:/var/lib/docker
  
  builder:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
      - DOCKER_BUILDKIT=1
      - COMPOSE_DOCKER_CLI_BUILD=1
    volumes:
      - ./project:/project
    working_dir: /project
    command: docker-compose build

volumes:
  docker-data:

BuildKit provides improved caching and parallel build execution.

Running Tests in Isolated Environments

Create isolated test environments:

version: '3.8'

services:
  test-docker:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - test-docker-data:/var/lib/docker
  
  test-runner:
    image: docker/compose:latest
    depends_on:
      - test-docker
    environment:
      - DOCKER_HOST=tcp://test-docker:2375
    volumes:
      - ./tests:/tests
      - ./app:/app
    working_dir: /tests
    command: sh -c "
      cd /app && docker-compose -f docker-compose.test.yml up --abort-on-container-exit &&
      docker-compose -f docker-compose.test.yml down -v
      "

volumes:
  test-docker-data:

Each test run uses a fresh Docker environment.

Parallel Build Environments

Run multiple isolated build environments:

version: '3.8'

services:
  dind-build-1:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data-1:/var/lib/docker
  
  dind-build-2:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data-2:/var/lib/docker
  
  builder-1:
    image: docker/compose:latest
    depends_on:
      - dind-build-1
    environment:
      - DOCKER_HOST=tcp://dind-build-1:2375
    volumes:
      - ./project-a:/workspace
    working_dir: /workspace
    command: docker-compose build
  
  builder-2:
    image: docker/compose:latest
    depends_on:
      - dind-build-2
    environment:
      - DOCKER_HOST=tcp://dind-build-2:2375
    volumes:
      - ./project-b:/workspace
    working_dir: /workspace
    command: docker-compose build

volumes:
  docker-data-1:
  docker-data-2:

Multiple projects build simultaneously in isolated environments.

Accessing DinD from Host

Configure DinD to be accessible from the host:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    ports:
      - "2375:2375"
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
    command: ["dockerd", "--host=tcp://0.0.0.0:2375"]

volumes:
  docker-data:

From the host:

export DOCKER_HOST=tcp://localhost:2375
docker ps
docker-compose up

Persistent DinD Data

Preserve images and containers across restarts:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
      - docker-cache:/root/.docker
    restart: unless-stopped

volumes:
  docker-data:
    driver: local
  docker-cache:
    driver: local

Named volumes persist DinD state between container restarts.

Registry Integration with DinD

Connect DinD to private registries:

version: '3.8'

services:
  registry:
    image: registry:2
    ports:
      - "5000:5000"
    volumes:
      - registry-data:/var/lib/registry
  
  dind:
    image: docker:24-dind
    privileged: true
    depends_on:
      - registry
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
    command: ["dockerd", "--insecure-registry=registry:5000"]
  
  builder:
    image: docker/compose:latest
    depends_on:
      - dind
      - registry
    environment:
      - DOCKER_HOST=tcp://dind:2375
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: sh -c "
      docker-compose build &&
      docker tag myapp:latest registry:5000/myapp:latest &&
      docker push registry:5000/myapp:latest
      "

volumes:
  registry-data:
  docker-data:

Build and push images to a containerized registry.

GitLab CI-like Environment

Simulate GitLab CI with DinD:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=/certs
    volumes:
      - docker-certs:/certs
      - docker-data:/var/lib/docker
  
  gitlab-runner-simulation:
    image: docker:24
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2376
      - DOCKER_CERT_PATH=/certs/client
      - DOCKER_TLS_VERIFY=1
      - CI=true
      - CI_PROJECT_DIR=/builds
    volumes:
      - docker-certs:/certs:ro
      - ./project:/builds
    working_dir: /builds
    command: sh -c "
      docker-compose -f docker-compose.ci.yml build &&
      docker-compose -f docker-compose.ci.yml run --rm test
      "

volumes:
  docker-certs:
  docker-data:

Layer Caching in DinD

Optimize build caching:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
      - DOCKER_BUILDKIT=1
    volumes:
      - docker-data:/var/lib/docker
      - buildkit-cache:/root/.cache/buildkit
  
  builder:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
      - DOCKER_BUILDKIT=1
      - COMPOSE_DOCKER_CLI_BUILD=1
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose build --parallel

volumes:
  docker-data:
  buildkit-cache:

Persistent cache volumes improve build performance.

Matrix Testing with DinD

Test against multiple configurations:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
  
  test-node-16:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
      - NODE_VERSION=16
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose -f docker-compose.test.yml run test
  
  test-node-18:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
      - NODE_VERSION=18
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose -f docker-compose.test.yml run test
  
  test-node-20:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
      - NODE_VERSION=20
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose -f docker-compose.test.yml run test

volumes:
  docker-data:

Run tests against multiple versions simultaneously.

Security Considerations for Socket Mounting

When mounting the Docker socket:

version: '3.8'

services:
  # DANGEROUS: Full Docker access
  unrestricted:
    image: docker:24
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
  
  # BETTER: Read-only socket access
  restricted:
    image: docker:24
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    command: docker ps  # Can only read, not modify

Read-only access limits potential security risks.

DinD with Custom Daemon Configuration

Configure the Docker daemon in DinD:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
      - ./daemon.json:/etc/docker/daemon.json:ro
    command: ["dockerd", "--config-file=/etc/docker/daemon.json"]

volumes:
  docker-data:

daemon.json:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  },
  "max-concurrent-downloads": 10
}

Compose with Multiple DinD Instances

Run completely isolated Docker environments:

version: '3.8'

services:
  dev-docker:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - dev-docker-data:/var/lib/docker
  
  staging-docker:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - staging-docker-data:/var/lib/docker
  
  dev-app:
    image: docker/compose:latest
    depends_on:
      - dev-docker
    environment:
      - DOCKER_HOST=tcp://dev-docker:2375
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose -f docker-compose.dev.yml up
  
  staging-app:
    image: docker/compose:latest
    depends_on:
      - staging-docker
    environment:
      - DOCKER_HOST=tcp://staging-docker:2375
    volumes:
      - ./app:/workspace
    working_dir: /workspace
    command: docker-compose -f docker-compose.staging.yml up

volumes:
  dev-docker-data:
  staging-docker-data:

Development and staging run in completely separate Docker instances.

Handling Networking in DinD

Configure networking for DinD containers:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
    command: ["dockerd", "--bip=172.30.0.1/16"]
  
  app:
    image: docker/compose:latest
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
    volumes:
      - ./project:/workspace
    working_dir: /workspace
    command: docker-compose up

volumes:
  docker-data:

The --bip flag sets the Docker bridge IP, avoiding conflicts.

Image Export and Import

Transfer images between DinD instances:

version: '3.8'

services:
  dind-source:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data-source:/var/lib/docker
      - image-transfer:/transfer
  
  dind-target:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data-target:/var/lib/docker
      - image-transfer:/transfer
  
  exporter:
    image: docker:24
    depends_on:
      - dind-source
    environment:
      - DOCKER_HOST=tcp://dind-source:2375
    volumes:
      - image-transfer:/transfer
    command: sh -c "
      docker save myapp:latest -o /transfer/myapp.tar
      "
  
  importer:
    image: docker:24
    depends_on:
      - dind-target
      - exporter
    environment:
      - DOCKER_HOST=tcp://dind-target:2375
    volumes:
      - image-transfer:/transfer
    command: sh -c "
      docker load -i /transfer/myapp.tar
      "

volumes:
  docker-data-source:
  docker-data-target:
  image-transfer:

Monitoring DinD Performance

Track DinD daemon performance:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
  
  monitor:
    image: docker:24
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
    command: sh -c "
      while true; do
        echo '=== Docker Info ==='
        docker info
        echo '=== Container Stats ==='
        docker stats --no-stream
        sleep 30
      done
      "

volumes:
  docker-data:

Cleanup and Maintenance

Implement automatic cleanup in DinD:

version: '3.8'

services:
  dind:
    image: docker:24-dind
    privileged: true
    environment:
      - DOCKER_TLS_CERTDIR=
    volumes:
      - docker-data:/var/lib/docker
  
  cleaner:
    image: docker:24
    depends_on:
      - dind
    environment:
      - DOCKER_HOST=tcp://dind:2375
    command: sh -c "
      while true; do
        docker system prune -af --volumes
        sleep 3600
      done
      "

volumes:
  docker-data:

Regular cleanup prevents disk space exhaustion.

Best Practices for DinD

Use named volumes for persistence: Store Docker data in named volumes to preserve images and containers across restarts.

Enable BuildKit: Always use BuildKit for better build performance and caching.

Implement cleanup: Regular cleanup prevents disk space issues in long-running DinD environments.

Use TLS for production: Enable TLS certificates for secure communication in production environments.

Limit resource usage: Set appropriate limits on DinD containers to prevent resource exhaustion.

Monitor disk usage: DinD can consume significant disk space; implement monitoring and alerting.

Use specific image tags: Avoid latest tags for DinD images to ensure consistent behavior.

Separate concerns: Use dedicated DinD instances for different purposes (builds, tests, deployments).

Consider alternatives: Evaluate if Docker socket mounting meets your needs before implementing full DinD.

Document security implications: Clearly document the security model and access controls for your DinD setup.

Test in isolation: Verify DinD setups work correctly before integrating into critical workflows.

Implement health checks: Monitor DinD daemon health to detect issues early.

Docker-in-Docker enables powerful containerized build systems, isolated testing environments, and sophisticated CI/CD pipelines. By understanding both true DinD and Docker socket mounting approaches, implementing proper security controls, managing networking and storage effectively, and following best practices for cleanup and monitoring, you can build robust containerized Docker environments that support complex development and deployment workflows.