Docker Compose

Compose turns a multi-container application into a declarative project—one YAML file describes services, networks, volumes, and secrets; docker compose up materializes the whole stack. Master the Compose Spec and you can reproduce any environment from laptop to single-server production without rewriting orchestration logic.

developer devops architect Compose V2 Compose Spec

Compose architecture

Compose is not a separate runtime—it is a client that drives the Docker Engine API. It reads YAML, resolves variables, merges files, and creates labeled resources grouped under a project name. Every container, network, and volume gets com.docker.compose.* labels so docker compose down can tear down exactly what up created.

compose.yaml vs docker-compose.yml

Both filenames work. Docker Compose searches, in order: compose.yaml, compose.yml, docker-compose.yaml, docker-compose.yml. The modern convention is compose.yaml—shorter, tool-agnostic, and aligned with the Compose Specification. Legacy repos still use docker-compose.yml; migrate when you touch the file anyway.

The Compose Specification

The Compose Spec is an open, vendor-neutral schema for multi-container apps. Docker Compose V2, Podman Compose, and other implementations target the same fields: services, networks, volumes, configs, secrets. Fields like deploy are implementation-specific—honored by Swarm and some production Compose setups, ignored by default local Engine.

docker compose V2 vs docker-compose V1 (deprecated)

Aspect docker compose (V2) docker-compose (V1)
Distribution Docker CLI plugin (docker compose) Standalone Python binary
Engine API Native BuildKit integration, compose watch Legacy build path; no watch
Spec support Compose Spec 3.x+ (no top-level version: required) Frozen at Compose file format v3; unmaintained
Status Current default (Docker Desktop, Engine 20.10+) Deprecated June 2023; remove from CI images
bash
# Verify V2 plugin (note: no hyphen)
docker compose version

# V1 still on PATH? Uninstall or alias away
which docker-compose  # should be empty or a warning wrapper

Project name

Compose groups resources under a project name, defaulting to the directory basename (folder my-api → project my-api). Override with COMPOSE_PROJECT_NAME, -p myproject, or top-level name: in compose.yaml. Project name prefixes container names (my-api-api-1) and default network (my-api_default).

Resource types

Resource Purpose Engine object
service A container definition (image or build, env, ports, deps) Container(s) — one per replica
network Private DNS + L3 connectivity between services Bridge/overlay network
volume Persistent or shared storage Named volume or bind mount
config Non-secret config file mounted into containers Swarm config / file bind (local)
secret Sensitive data (passwords, keys) — not in env vars Swarm secret / file in /run/secrets
flowchart LR YAML["compose.yaml"] --> CLI["docker compose"] ENV[".env / shell vars"] --> CLI CLI --> API["Docker Engine API"] API --> SVC["Containers"] API --> NET["Networks"] API --> VOL["Volumes"]
🔬 Under the Hood

Compose V2 is written in Go and ships as a CLI plugin. It builds an internal model from merged YAML, then issues the same API calls you would script with docker container create, network create, and volume create—with consistent labels for project tracking.

⚠️ Pitfall

Running two clones of the same repo in different directories creates two projects with separate networks and volumes. Port collisions happen when both publish 8080:8080. Use -p or per-developer .env overrides for host ports.

🎯 Interview Tip

When asked "Docker vs Compose," say: Docker runs one container; Compose orchestrates a labeled project of many containers with shared networking and lifecycle commands. Compose does not replace Kubernetes—it solves single-host, declarative multi-container workflows.

compose.yaml deep dive

Every production-grade compose file balances build vs pull, startup ordering, and network isolation. The fields below are the ones you will edit weekly—not the full spec appendix.

Minimal project skeleton

yaml
name: order-platform

services:
  api:
    image: ghcr.io/acme/api:${API_TAG:-latest}
    ports:
      - "${API_PORT:-8080}:8080"
    environment:
      DATABASE_URL: postgres://app:${DB_PASSWORD}@db:5432/orders
    depends_on:
      db:
        condition: service_healthy
    networks:
      - frontend
      - backend

  db:
    image: postgres:16-alpine
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d orders"]
      interval: 5s
      timeout: 3s
      retries: 5
    networks:
      - backend

networks:
  frontend:
  backend:
    internal: true

volumes:
  pgdata:

image vs build

Use image: when CI already built and pushed the artifact—you want reproducible digests in prod. Use build: for local iteration where source changes should trigger rebuilds. You can specify both: Compose tags the build result with the image name.

build: context, args, target

yaml
services:
  api:
    build:
      context: ./services/api          # directory sent as build context
      dockerfile: Dockerfile.prod    # relative to context
      args:
        NODE_ENV: development
        APP_VERSION: ${GIT_SHA:-dev}
      target: runtime                # multi-stage: stop at this stage
    image: acme/api:${GIT_SHA:-dev}

ports, volumes, environment

Field Syntax Notes
ports "HOST:CONTAINER" or "127.0.0.1:8080:8080" Long syntax supports published, target, protocol
volumes named_vol:/path or ./host:/container Named volumes survive down; bind mounts reflect host files live
environment Map or list (KEY=value) Prefer env_file: for long lists; never commit secrets

depends_on with conditions

Plain depends_on: [db] only waits for the container to start, not for Postgres to accept connections. Use condition with healthchecks for real readiness:

yaml
services:
  api:
    depends_on:
      db:
        condition: service_healthy
      redis:
        condition: service_started   # no healthcheck defined

  db:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U $$POSTGRES_USER"]
      interval: 5s
      retries: 10
      start_period: 30s

healthcheck

Healthchecks run inside the container. Use CMD-SHELL for shell pipelines, CMD for exec form. start_period suppresses failure counts during slow boot (migrations, JVM warmup).

networks

Services join networks listed under networks:. DNS name = service name. internal: true blocks outbound internet—good for databases and message brokers that should only talk to app tiers.

deploy block

The deploy key configures Swarm mode and is ignored by default on plain Docker Engine. Some teams use docker compose --compatibility to map replicas and limits to Engine. For single-host prod without Swarm, set mem_limit, cpus, and restart at service level instead.

yaml
services:
  worker:
    image: acme/worker:2.1.0
    deploy:
      replicas: 2
      resources:
        limits:
          cpus: "0.50"
          memory: 512M
        reservations:
          memory: 256M
      restart_policy:
        condition: on-failure
        max_attempts: 3

profiles

Profiles gate optional services. Only services without a profile, or whose profile you activate, start on up.

bash
# Start core stack only
docker compose up -d

# Add observability and mail catcher
docker compose --profile observability --profile devtools up -d
yaml
services:
  prometheus:
    image: prom/prometheus:v2.52.0
    profiles: [observability]

  mailhog:
    image: mailhog/mailhog
    profiles: [devtools]

extends (legacy pattern)

extends let one service inherit another's definition from a different file. The Compose Spec deprecated extends in favor of include (Compose 2.20+) or YAML anchors. Prefer multiple -f merges or shared fragments via include:.

💡 Pro Tip

Run docker compose config before every PR that touches YAML—it resolves variables, merges overrides, and prints the effective file. Catch typos and circular dependencies before CI does.

⚙️ Config

Drop the top-level version: "3.8" key in new files. Compose V2 infers the schema from the Spec; the version field is obsolete and confuses reviewers.

⚖️ Trade-off

build: in compose.yaml is convenient locally but couples deploy artifacts to developer laptops. Architects often use image: with immutable tags in prod compose files and keep build: only in override files.

Multi-environment compose

One repository, three runtimes: laptop, staging VM, production server. The pattern is a base compose.yaml plus environment-specific overlays—never three unrelated copies that drift apart.

File layout

text
deploy/
├── compose.yaml           # shared services, networks, volumes
├── compose.override.yaml  # auto-loaded locally (gitignored or committed)
├── compose.prod.yaml      # prod: image tags, limits, no bind mounts
├── .env.example           # documented defaults (committed)
└── .env                   # local secrets (gitignored)

Base compose.yaml

yaml
name: order-platform

services:
  api:
    image: ghcr.io/acme/api:${API_TAG:-latest}
    environment:
      LOG_LEVEL: ${LOG_LEVEL:-info}
      DATABASE_URL: postgres://app:${DB_PASSWORD}@db:5432/orders
    depends_on:
      db:
        condition: service_healthy
    networks: [app]

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: orders
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d orders"]
      interval: 5s
      retries: 5
    networks: [app]

networks:
  app:

volumes:
  pgdata:

compose.override.yaml (local dev)

Compose automatically merges compose.override.yaml when present—no -f flag needed. Override files typically add bind mounts, published ports, and debug env vars.

yaml
services:
  api:
    build:
      context: ../services/api
      target: dev
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ../services/api/src:/app/src:cached
    environment:
      LOG_LEVEL: debug
      DEBUG: "true"

compose.prod.yaml

yaml
services:
  api:
    image: ghcr.io/acme/api:${API_TAG}   # required in prod — no default
    restart: unless-stopped
    ports:
      - "443:8080"
    deploy:
      resources:
        limits:
          memory: 1G
          cpus: "1.0"
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"

  db:
    restart: unless-stopped
    # no host port publish — DB reachable only on app network

Merging with -f

Later files override earlier keys. Production deploy:

bash
docker compose \
  -f compose.yaml \
  -f compose.prod.yaml \
  --env-file .env.prod \
  up -d

Variable substitution: ${VAR:-default}

Syntax Behavior
${VAR} Required; empty string if unset (may break startup)
${VAR:-default} Use default when unset or empty
${VAR:?error} Fail fast with message if unset—ideal for prod secrets

.env file

Compose loads .env from the project directory for interpolation only—it does not automatically inject vars into containers unless referenced in environment: or env_file:. Precedence: shell env > --env-file > .env.

env
# .env.example — commit this
API_TAG=latest
API_PORT=8080
LOG_LEVEL=info
DB_PASSWORD=change-me-local

# .env — gitignore; copy from example
DB_PASSWORD=dev-secret-42
🔒 Security

Never commit .env with production credentials. Use ${DB_PASSWORD:?set DB_PASSWORD} in prod overlays so a missing secret aborts deploy instead of starting with empty passwords.

📦 Real World

Teams on GitOps often generate the final compose from templates (Helm-style) at deploy time. The source of truth remains the base + overlay pattern; CI injects image digests and secrets from a vault.

Compose for local development

Local Compose should optimize for fast feedback: edit code on the host, see changes in the container, attach a debugger, seed realistic data. Production concerns (immutable images, tight resource caps) stay in overlays.

Bind mounts for hot reload

Mount source directories over the container path so interpreters with watch mode (Node, Go air, Spring DevTools) reload without rebuilds. Use :cached on macOS Docker Desktop to reduce filesystem sync overhead.

yaml
services:
  api:
    build:
      context: ./api
      target: dev
    command: npm run dev
    volumes:
      - ./api/src:/app/src:cached
      - ./api/package.json:/app/package.json:ro
    environment:
      NODE_ENV: development
      CHOKIDAR_USEPOLLING: "true"   # file watch on Docker Desktop VM

Debugger entrypoint override

Override command or entrypoint in an override file to start the JVM or Node inspector listening on a published port.

yaml
services:
  api:
    ports:
      - "8080:8080"
      - "9229:9229"   # Node inspector
    command: node --inspect=0.0.0.0:9229 src/index.js

  # Java example
  billing:
    ports:
      - "5005:5005"
    environment:
      JAVA_TOOL_OPTIONS: "-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005"

Database seeding

Postgres and MySQL official images run /docker-entrypoint-initdb.d/* scripts on first volume init. Mount SQL or shell seeders for consistent local data.

yaml
services:
  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: orders
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./db/init:/docker-entrypoint-initdb.d:ro   # 01-schema.sql, 02-seed.sql

volumes:
  pgdata:

Full local stack example

yaml
name: shop-local

services:
  api:
    build:
      context: ./api
      target: dev
    ports:
      - "127.0.0.1:3000:3000"
    volumes:
      - ./api:/app:cached
    environment:
      REDIS_URL: redis://cache:6379
      DATABASE_URL: postgres://app:dev@db:5432/shop
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started
    networks: [frontend, backend]

  db:
    image: postgres:16-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD: dev
      POSTGRES_DB: shop
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./db/init:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d shop"]
      interval: 3s
      retries: 10
    networks: [backend]

  cache:
    image: redis:7-alpine
    networks: [backend]

  worker:
    build:
      context: ./worker
    command: npm run jobs:dev
    volumes:
      - ./worker:/app:cached
    environment:
      REDIS_URL: redis://cache:6379
    depends_on: [cache]
    networks: [backend]
    profiles: [jobs]

networks:
  frontend:
  backend:

volumes:
  pgdata:

docker compose watch (sync / rebuild)

Compose Watch (V2.22+) watches host paths and syncs files into running containers or rebuilds on change—no manual volume gymnastics for some workflows.

yaml
services:
  api:
    build: ./api
    develop:
      watch:
        - path: ./api/src
          action: sync
          target: /app/src
        - path: ./api/package.json
          action: rebuild
bash
docker compose watch
# or
docker compose up --watch
💡 Pro Tip

Add an anonymous volume over node_modules (/app/node_modules) when bind-mounting a Node app—otherwise host node_modules (wrong arch) shadows the container install.

⚠️ Pitfall

Bind-mounting over a directory that also contains build artifacts can hide files the image expects. Use multi-stage Dockerfiles with a dev target that installs deps inside the image, then mount only src/.

Compose networking

Every Compose project gets a default bridge network with embedded DNS. Service name db resolves to the DB container IP—no hard-coded addresses. Tiered networks add defense in depth; external networks connect to existing infrastructure.

Default project network

If you omit networks:, all services join <project>_default. Containers reach each other by service name on any internal port—only ports: publishes to the host.

bash
# From api container — DNS resolves "db"
docker compose exec api getent hosts db

# Inspect the project network
docker network inspect shop-local_backend

Custom tier networks

Split frontend (edge + API) from backend (API + data stores). Mark backend internal: true to block outbound internet while preserving east-west traffic.

yaml
services:
  gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    networks: [frontend]

  api:
    image: acme/api:1.4.0
    networks: [frontend, backend]

  db:
    image: postgres:16-alpine
    networks: [backend]

networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge
    internal: true

External networks

Join a network created outside the project—shared reverse proxy, monitoring agent, or legacy VM bridge. Compose will not delete external networks on down.

bash
# Create once on the host
docker network create proxy

# compose.yaml references it
yaml
services:
  api:
    image: acme/api:1.4.0
    networks:
      - default
      - proxy

networks:
  proxy:
    external: true
    name: proxy
🔬 Under the Hood

Compose uses the embedded DNS server at 127.0.0.11 inside each container. It returns A records for service names scoped to networks that container shares—if api is only on frontend, a container on backend-only cannot resolve it.

⚖️ Trade-off

Host networking (network_mode: host) removes NAT overhead but breaks cross-platform portability and DNS-based service discovery. Reserve for performance-sensitive agents on Linux prod servers—not for default dev stacks.

Compose commands reference

V2 commands live under docker compose (space, not hyphen). Global flags: -f (files), -p (project), --profile, --env-file.

Command Purpose Common flags / example
up Create and start services docker compose up -d --build --remove-orphans
down Stop and remove containers, networks; optional volumes docker compose down -v --rmi local
logs Stream aggregated service logs docker compose logs -f --tail=100 api worker
exec Run command in running container docker compose exec api sh
ps List project containers and state docker compose ps -a
build Build or rebuild images docker compose build --no-cache api
pull Pull service images docker compose pull
scale Scale service replicas (non-Swarm) docker compose up -d --scale worker=3
run One-off command in new container docker compose run --rm api npm test
config Validate and print merged compose docker compose config --quiet
top Show running processes per service docker compose top api

Day-one workflow

bash
docker compose config          # validate YAML + env
docker compose pull            # refresh base images
docker compose up -d --build   # build local services, start detached
docker compose ps              # health status
docker compose logs -f api     # tail one service
docker compose exec db psql -U app -d orders
docker compose run --rm api npm test
docker compose down            # stop; add -v to wipe named volumes
💡 Pro Tip

docker compose run does not publish ports by default and creates a one-off container—use it for migrations and tests. exec requires the service to already be running.

🎯 Interview Tip

Know the difference: down -v deletes named volumes declared in the compose file—data loss in dev if you forget. stop halts containers but keeps them for inspection.

Production compose patterns

Compose on a single VM or bare-metal server is a valid production model for small teams—especially before Kubernetes ROI makes sense. Treat the compose file like infrastructure code: pinned digests, secrets, health gates, and log rotation.

Single-server production compose.yaml

yaml
name: order-platform-prod

services:
  api:
    image: ghcr.io/acme/api@sha256:abc123...   # immutable digest
    restart: unless-stopped
    ports:
      - "443:8080"
    environment:
      DATABASE_URL: postgres://app@db:5432/orders
      LOG_LEVEL: info
    secrets:
      - db_password
    depends_on:
      db:
        condition: service_healthy
    deploy:
      resources:
        limits:
          cpus: "2.0"
          memory: 2G
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 60s
    logging:
      driver: json-file
      options:
        max-size: "25m"
        max-file: "10"
    networks: [edge, data]

  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: app
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
      POSTGRES_DB: orders
    secrets:
      - db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d orders"]
      interval: 10s
      retries: 5
    deploy:
      resources:
        limits:
          memory: 4G
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "5"
    networks: [data]

secrets:
  db_password:
    file: ./secrets/db_password.txt   # chmod 600; not in git

networks:
  edge:
  data:
    internal: true

volumes:
  pgdata:
    driver: local

Pattern checklist

Pattern Why Compose mechanism
Resource limits Prevent one container OOM-killing the host deploy.resources.limits or mem_limit / cpus
Healthchecks + depends_on Avoid traffic to half-ready APIs condition: service_healthy
Secrets Keep credentials out of env and image layers secrets:/run/secrets/
restart: unless-stopped Survive daemon restarts and reboots Service-level restart policy
Log rotation json-file logs fill disk without limits logging.options.max-size / max-file
No bind mounts Immutable deploy surface Named volumes only; images from CI

Deploy command

bash
# On production host
export API_TAG=sha256:abc123...
docker compose -f compose.yaml -f compose.prod.yaml pull
docker compose -f compose.yaml -f compose.prod.yaml up -d --remove-orphans
docker compose ps
🔒 Security

Mount the Docker socket (/var/run/docker.sock) only in trusted sidecars. Any container with socket access has effective root on the host. Prefer read-only root filesystem and dropped capabilities where your images support it.

⚖️ Trade-off

Compose vs Kubernetes: Compose excels at 1–3 node deployments with simple rollouts. When you need zero-downtime rolling updates across many nodes, pod autoscaling, or CRDs—migrate the same images to K8s. The compose file remains valuable as documentation of service relationships.

📦 Real World

Many teams run Traefik or Caddy as a separate compose project on an external proxy network. App stacks join that network; TLS termination stays in the edge stack while app compose files stay portable.