Docker & Docker Compose Deep Dive: Multi-Stage Builds, Layer Caching & Production Optimization

Why Docker Image Size Matters in Production

A naive Django Docker image built from python:3.12 with all build tools included weighs around 1.2 GB. A properly optimised multi-stage image serving the exact same application weighs 78 MB — a 15x reduction meaning 15x faster deploys, 15x less storage on your container registry, and dramatically reduced attack surface.

This guide walks through the complete Docker production playbook: multi-stage builds, BuildKit layer caching, secrets management, health checks, and a production-grade docker-compose.yml that mirrors real industry setups.

1. Multi-Stage Builds — The Foundation

Multi-stage builds allow multiple FROM statements in a single Dockerfile. Only the final stage ends up in your shipped image — previous stages are discarded after the build completes.

# BAD — everything in one stage, 1.2GB image
FROM python:3.12
RUN apt-get update && apt-get install -y gcc libpq-dev build-essential
WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt
COPY . .
CMD ["gunicorn", "reqpulse.wsgi:application", "--bind", "0.0.0.0:8000"]

# GOOD — multi-stage, 78MB final image
FROM python:3.12-slim AS builder
RUN apt-get update && apt-get install -y gcc libpq-dev && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY requirements.txt .
RUN pip install --upgrade pip &&     pip wheel --no-cache-dir --wheel-dir /wheels -r requirements.txt

FROM python:3.12-slim AS runtime
RUN apt-get update && apt-get install -y libpq5 && rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY --from=builder /wheels /wheels
RUN pip install --no-cache-dir --no-index --find-links=/wheels /wheels/*.whl
COPY . .
RUN python manage.py collectstatic --noinput
EXPOSE 8000
CMD ["gunicorn", "reqpulse.wsgi:application",      "--bind", "0.0.0.0:8000", "--workers", "4", "--timeout", "120"]

Size Comparison

2. Layer Caching — Maximising Rebuild Speed

Docker builds each instruction as an immutable layer. The golden rule: put things that change rarely at the top, things that change often at the bottom.

# BAD cache ordering
COPY . .                           # Cache busted every code change
RUN pip install -r requirements.txt   # Re-installs everything each time

# GOOD cache ordering
COPY requirements.txt .            # Only bust cache when requirements change
RUN pip install -r requirements.txt
COPY . .                           # Source change doesn't re-install deps

BuildKit Cache Mounts (Docker 18.09+)

# syntax=docker/dockerfile:1
FROM python:3.12-slim AS builder

RUN --mount=type=cache,target=/root/.cache/pip     pip install --upgrade pip

COPY requirements.txt .
RUN --mount=type=cache,target=/root/.cache/pip     pip wheel --wheel-dir /wheels -r requirements.txt

export DOCKER_BUILDKIT=1

3. Secrets Management — Never Bake Credentials Into Images

Security rule: If your secret ever appears in a Docker layer — even if deleted in the next RUN instruction — it is permanently embedded in the image history and can be extracted with docker image history.

# syntax=docker/dockerfile:1
FROM python:3.12-slim
RUN --mount=type=secret,id=pip_token     pip install private-package     --extra-index-url "https://token:$(cat /run/secrets/pip_token)@pypi.company.com"

echo "$PYPI_TOKEN" | docker build   --secret id=pip_token,src=/dev/stdin   -t myapp:latest .

4. Production docker-compose.yml

version: "3.9"

x-django-env: &django-env
  DATABASE_URL: postgres://postgres:secret@db:5432/mydb
  REDIS_URL: redis://redis:6379/0
  SECRET_KEY: ${SECRET_KEY}
  DEBUG: "False"

services:
  db:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_DB: mydb
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: secret
    volumes:
      - postgres_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 10s
      timeout: 5s
      retries: 5

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server --maxmemory 128mb --maxmemory-policy allkeys-lru

  web:
    build:
      context: .
      target: runtime
    restart: unless-stopped
    environment:
      <<: *django-env
    depends_on:
      db:
        condition: service_healthy
    expose:
      - "8000"

  nginx:
    image: nginx:alpine
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro

volumes:
  postgres_data:

5. .dockerignore — Critical for Fast Builds

.git/
*.pyc
__pycache__/
.pytest_cache/
.venv/
env/
node_modules/
.env
.env.*
*.log
*.sqlite3
README.md
docs/
.agents/

Without .dockerignore, every COPY . . sends your entire project including .git and node_modules to the Docker daemon — drastically slowing build context transfer on large repositories.

6. Health Checks and Graceful Shutdown

# health/views.py
from django.http import JsonResponse
from django.db import connection

def health_check(request):
    try:
        connection.ensure_connection()
        db_ok = True
    except Exception:
        db_ok = False
    status = 200 if db_ok else 503
    return JsonResponse({"status": "ok" if db_ok else "degraded", "db": db_ok}, status=status)

Key insight: Always add STOPSIGNAL SIGTERM to your Dockerfile and handle SIGTERM in your application. Kubernetes sends SIGTERM before SIGKILL, giving your app 30 seconds to finish in-flight requests.