Multi-Stage Docker Builds — Secrets Survive rm

Every second your Docker image takes to pull across a network is a second your deployment is stalled. In Kubernetes environments rolling out hundreds of pods under load, or CI pipelines building dozens of images a day, bloated images are a reliability and cost problem. A Node.js app shipping with its full devDependencies, TypeScript compiler, and build toolchain alongside the production binary is a 1.2 GB image waiting to become a 3 AM outage.

Traditional single-stage Dockerfiles are all-or-nothing. You install the compiler, build the binary, copy the source — and all of it ends up baked into the final layer. Docker does not have a native concept of 'clean up after yourself' within a single build context, because every RUN instruction adds a new immutable layer. Removing files in a later layer does not reclaim space — it just hides them.

Multi-stage builds solve this by introducing multiple isolated build contexts inside one Dockerfile. Each FROM starts a fresh stage with its own filesystem. Only artifacts you explicitly copy forward survive into the final image. The build tools, intermediate object files, and source code are discarded with the build stage. This is the single most impactful Dockerfile optimization for production images.

Why rm in a Dockerfile Doesn't Shrink Your Image

Multi-stage Docker builds let you compile code in one stage and copy only the final artifact into a minimal runtime image. The core mechanic: multiple FROM statements in a single Dockerfile, each starting a new stage, and COPY --from=target to extract files from earlier stages. This eliminates the need to chain RUN apt-get remove or rm commands that never actually reduce image size because each RUN layer is additive — deleting a file in a later layer only marks it as hidden, not removed. The final image still contains the full weight of every intermediate layer.

A typical Java workflow: stage one uses a full JDK image (e.g., eclipse-temurin:17-jdk) to compile and package a Spring Boot fat JAR. Stage two starts from a slim JRE image (e.g., eclipse-temurin:17-jre-alpine) and copies only the JAR from stage one. The resulting image is ~180 MB instead of ~400 MB. No apt-get purge, no rm -rf /var/lib/apt/lists/* — just a clean COPY. Each stage can use different base images, different working directories, and even different OS families.

Use multi-stage builds whenever your build process requires tools absent from the runtime image — compilers, build tools, test runners. This is every production Java service, every microservice, every CI pipeline. The alternative is either a bloated image with build-time dependencies or a fragile script that tries to clean up after itself. Multi-stage is the only pattern that guarantees a minimal, reproducible runtime image without sacrificing build-time tooling.

How Multi-Stage Builds Work at the Layer Level

A Dockerfile with multiple FROM instructions creates multiple isolated stages. Each stage starts with a fresh filesystem initialised from its base image. Stages are identified by their index (0, 1, 2...) or by an alias assigned with AS.

The critical insight: only files you explicitly COPY --from=<stage> are transferred between stages. Everything else — compilers, build caches, intermediate object files, source code — exists only in the build stage's filesystem and is discarded when the build completes. The final image contains only the last stage's filesystem plus any files copied into it.

Docker's layer system means each RUN, COPY, and ADD instruction creates an immutable layer. In a single-stage build, RUN rm file creates a NEW layer that hides the file — the original layer with the file still exists in the image. Multi-stage builds avoid this entirely by never including the file in the final stage's layers in the first place.

# ─────────────────────────────────────────────────────────────
# Stage 1: Build stage — contains Go compiler, source, dependencies
# This stage is ~800 MB but NEVER ships to production
# ─────────────────────────────────────────────────────────────
FROM golang:1.22-alpine AS builder

WORKDIR /app

# Copy dependency files FIRST — this layer is cached until go.mod changes
COPY go.mod go.sum ./
RUN go mod download

# Copy source code AFTER dependencies — changing source doesn't invalidate dep cache
COPY . .

# Build a statically linked binary — no runtime dependencies needed
# CGO_ENABLED=0 ensures no C library dependency
# -ldflags='-s -w' strips debug symbols, reducing binary size by ~30%
RUN CGO_ENABLED=0 GOOS=linux go build \
    -ldflags='-s -w' \
    -o /app/server \
    ./cmd/server

# ─────────────────────────────────────────────────────────────
# Stage 2: Runtime stage — contains ONLY the binary and config
# This stage is ~15-25 MB — a 97% reduction from the build stage
# ─────────────────────────────────────────────────────────────
FROM alpine:3.19 AS runtime

# Install CA certificates for HTTPS and tzdata for timezone support
RUN apk --no-cache add ca-certificates tzdata

WORKDIR /app

# Copy ONLY the compiled binary from the builder stage
# Everything else (Go compiler, source, build cache) is discarded
COPY --from=builder /app/server .

# Copy config files if needed
COPY --from=builder /app/config ./config

# Run as non-root user — security best practice
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

EXPOSE 8080

ENTRYPOINT ["./server"]

Stage Targeting with --target: Production vs Build vs Test

Docker allows you to stop the build at any named stage using the --target flag. This is invaluable for development workflows, CI, and debugging. Without --target, Docker builds all stages up to the last FROM. With --target, you can request only the stages you need.

# ─────────────────────────────────────────────────────────────
# Multi-stage Dockerfile with explicit stage targets
# Build with: docker build --target <stage> -t image .
# ─────────────────────────────────────────────────────────────

# Stage 0: Base dependencies (shared across build and dev)
FROM node:20-alpine AS base
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

# Stage 1: Development — includes dev dependencies and tools
FROM base AS development
RUN npm ci --include=dev
COPY . .
RUN npm install -g nodemon
CMD ["nodemon", "src/index.js"]

# Stage 2: Test — runs tests with coverage
FROM base AS test
RUN npm ci --include=dev
COPY . .
RUN npm test

# Stage 3: Build — compiles TypeScript, builds assets
FROM base AS build
COPY . .
RUN npm run build

# Stage 4: Production — minimal runtime image
FROM node:20-alpine AS production
WORKDIR /app
# Copy only production dependencies from base
COPY --from=base /app/node_modules ./node_modules
# Copy built application from build stage
COPY --from=build /app/dist ./dist
COPY --from=build /app/package.json ./
USER node
CMD ["node", "dist/index.js"]

BuildKit, Parallel Stages, and Cache Mounts

BuildKit is Docker's modern build engine, enabled by setting DOCKER_BUILDKIT=1 or using docker buildx. It brings three critical capabilities to multi-stage builds:

1. Parallel stage execution: Stages that do not depend on each other run concurrently. If your Dockerfile has a test stage and a build stage that both depend on the dependency-install stage, BuildKit runs test and build in parallel after dependencies are installed.
2. Cache mounts: --mount=type=cache persists a directory across builds without invalidating the layer. This is transformative for package managers — mount the npm/pip/go cache directory so dependency downloads are cached across builds even when the layer would otherwise be invalidated.
3. Secret mounts: --mount=type=secret provides a temporary file during RUN execution that is never stored in any layer. This is the correct way to use API keys, tokens, and credentials during builds.

# syntax=docker/dockerfile:1  # required for BuildKit features

# ─────────────────────────────────────────────────────────────
# Stage 1: Dependency installation with cache mount
# The npm cache persists across builds — re-installs are near-instant
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS deps

WORKDIR /app

COPY package.json package-lock.json ./

# --mount=type=cache persists /root/.npm across builds
# Even if this layer is invalidated, the cache is not lost
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

# ─────────────────────────────────────────────────────────────
# Stage 2: Build stage with secret mount for private registry
# The npm token is available during build but never stored in a layer
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS builder

WORKDIR /app

COPY package.json package-lock.json ./

# Secret mount: token is available as a file during this RUN only
# It is NEVER stored in any layer — not even in a hidden layer
RUN --mount=type=secret,id=npm_token,target=/run/secrets/npm_token \
    --mount=type=cache,target=/root/.npm \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

COPY . .

RUN npm run build

# ─────────────────────────────────────────────────────────────
# Stage 3: Production runtime — minimal image
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS runtime

WORKDIR /app

# Copy only production dependencies from deps stage
COPY --from=deps /app/node_modules ./node_modules

# Copy built application from builder stage
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./

# Run as non-root
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

EXPOSE 3000

CMD ["node", "dist/index.js"]

# ─────────────────────────────────────────────────────────────
# Build command with secret:
# DOCKER_BUILDKIT=1 docker build \
#   --secret id=npm_token,src=.npmrc_token \
#   -t myapp:latest .

Production Patterns: Go, Node.js, and Java

Each language ecosystem has specific multi-stage build patterns that address its unique characteristics. The patterns below are battle-tested in production and handle the most common failure modes.

# ─────────────────────────────────────────────────────────────
# Java Multi-Stage Build: Maven build + JRE runtime
# Reduces image from ~800 MB (JDK) to ~120 MB (JRE alpine)
# ─────────────────────────────────────────────────────────────

# Stage 1: Build with Maven
FROM maven:3.9-eclipse-temurin-21 AS builder

WORKDIR /app

# Copy POM first — this layer is cached until pom.xml changes
COPY pom.xml .

# Download dependencies with cache mount
RUN --mount=type=cache,target=/root/.m2 \
    mvn dependency:go-offline -B

# Copy source and build
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 \
    mvn package -DskipTests -B

# Stage 2: Extract JRE runtime
FROM eclipse-temurin:21-jre-alpine AS runtime

WORKDIR /app

# Copy only the fat JAR from the build stage
COPY --from=builder /app/target/*.jar app.jar

# JVM tuning for containers — critical for production
# -XX:+UseContainerSupport respects cgroup memory limits
# -XX:MaxRAMPercentage=75% uses 75% of container memory for heap
ENV JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC"

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

Image Size Reduction Comparison Table

One of the strongest arguments for multi-stage builds is the dramatic reduction in image size. Below is a comparison of typical image sizes for common language stacks using single-stage vs. optimal multi-stage builds. These numbers are based on production images (source code + dependencies + runtime) and assume best practices like using slim/alpine bases and stripping debug symbols.

The pattern is consistent: build tools and compilers account for 70-90% of image bulk. Multi-stage builds eliminate them from the final image, leaving only the runtime and the compiled artifacts.

Multi-Stage Build Example: Go (Minimal Pattern)

This section provides a focused, minimal multi-stage Dockerfile for a Go application. It demonstrates the canonical pattern: build stage containing the compiler and source, and a runtime stage with only the binary. The result is a 97% reduction in image size.

Key steps: 1. Use golang:alpine as the build stage – includes Go compiler and standard library, but is smaller than full debian-based images. 2. Copy go.mod/go.sum first – layer caching ensures go mod download runs only when dependencies change. 3. Build with CGO_ENABLED=0 – produces a statically linked binary that does not depend on glibc, allowing FROM scratch as runtime. 4. Use -ldflags='-s -w' – strips debug symbol tables, reducing binary size by ~30%. 5. Runtime base: scratch – zero bytes, no shell, no package manager. Maximum security and minimal size.

# syntax=docker/dockerfile:1
# Minimal Go multi-stage build

# Stage 1: Build the Go binary
FROM golang:1.22-alpine AS builder

WORKDIR /build

# Download dependencies first - leverages layer caching
COPY go.mod go.sum ./
RUN go mod download

# Copy source code
COPY . .

# Build a static binary, strip debug symbols
RUN CGO_ENABLED=0 GOOS=linux go build \
    -ldflags='-s -w' \
    -o /build/server \
    ./cmd/server

# Stage 2: Runtime - absolutely minimal
FROM scratch AS runtime

# Copy CA certificates for HTTPS calls (from builder stage)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Copy the binary
COPY --from=builder /build/server /server

# No shell, no package manager, no user (not needed - scratch has /dev/null etc.)
EXPOSE 8080

ENTRYPOINT ["/server"]

Security: Secret Management and Image Scanning

Multi-stage builds are a security primitive, not just a size optimisation. The build stage isolation means secrets used during compilation never appear in the final image — IF you use the correct mechanisms. The wrong mechanism (ENV, ARG, or inline RUN) leaks secrets into layers permanently.

Three rules for secret management in Docker builds: 1. Never use ENV or ARG for secrets — they persist in image metadata. 2. Never hardcode secrets in RUN commands — they persist in layer history. 3. Always use BuildKit secret mounts — --mount=type=secret provides temporary file access without layer persistence.

Beyond secrets, the final image should be scanned for vulnerabilities. Even a minimal alpine base image may contain packages with known CVEs. Trivy, Grype, and Snyk Container can scan images in CI and block deployment if critical vulnerabilities are found.

# syntax=docker/dockerfile:1

# ─────────────────────────────────────────────────────────────
# WRONG: Secrets in ENV or ARG — visible in docker history and inspect
# ─────────────────────────────────────────────────────────────
# DO NOT DO THIS:
# ENV NPM_TOKEN=ghp_xxxxxxxxxxxx           ← visible in docker inspect
# ARG DB_PASSWORD=secret123                ← visible in docker history
# RUN echo $NPM_TOKEN > .npmrc             ← visible in layer history

# ─────────────────────────────────────────────────────────────
# CORRECT: BuildKit secret mounts
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS builder

WORKDIR /app

COPY package.json package-lock.json ./

# Secret is mounted as a file at /run/secrets/npm_token
# Available only during this RUN command
# Not stored in any layer — not even hidden
RUN --mount=type=secret,id=npm_token,target=/run/secrets/npm_token \
    --mount=type=cache,target=/root/.npm \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

COPY . .
RUN npm run build

# ─────────────────────────────────────────────────────────────
# Final stage: no secrets, no build tools, no source code
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS runtime

WORKDIR /app

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./

USER node

CMD ["node", "dist/index.js"]

# ─────────────────────────────────────────────────────────────
# Build command:
# DOCKER_BUILDKIT=1 docker build \
#   --secret id=npm_token,src=./.npm_token_value \
#   -t myapp:latest .
#
# Verify no secrets in image:
# docker history myapp:latest | grep -i token  # should return nothing
# docker inspect myapp:latest | grep -i token   # should return nothing

Name Your Build Stages or Get Burned in Debugging

Most devs treat stage names like commit messages — useless until you need them. Don't be that person. Naming stages with descriptive aliases isn't for clarity in the Dockerfile. It's for survival when you're chasing a 3AM build failure.

When you reference --from=build instead of --from=0, you can read your CI logs without a decoder ring. But the real win comes from docker build --target=test. That single line lets you run integration tests in a throwaway stage without rebuilding the entire pipeline. Your production image stays untouched. Your CI minutes stay human.

Here's the rule: every stage that produces something you might inspect, test, or copy from needs a name. build, test, dev-deps, production. If it only exists to install compilers, leave it unnamed — it's dead code the moment it finishes.

The senior move? Use stage names as documentation. If I see FROM golang:1.21 AS vet-lint, I know exactly what that stage does. If I see FROM node:18 AS build, I know the author has been burned by FROM 0.

// io.thecodeforge — devops tutorial

# BAD: Unnamed stages, numeric references
FROM golang:1.21 AS 0
RUN go build -o /app/service .

FROM alpine:3.19
COPY --from=0 /app/service /service

# GOOD: Named stages, explicit targets
FROM golang:1.21 AS build-service
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o /app/service .

FROM golang:1.21 AS test
COPY --from=build-service /src /src
RUN go test ./... -v

FROM alpine:3.19 AS production
COPY --from=build-service /app/service /service
CMD ["/service"]

Go Distroless or Go Home: Why Scratch Isn't Enough

You've shrunk your Go binary from 800MB to 12MB using multi-stage builds. Congratulations. Now what? Your scratch image has no shell, no ls, no curl. A perfectly minimal attack surface. But also a perfectly useless debugging environment when production misbehaves.

Distroless images from Google (gcr.io/distroless/base) give you the middle ground: they contain essential runtime dependencies — glibc, SSL certificates, timezone data — but no package manager, no shell, no exploit-friendly utilities. Your 12MB Go binary on scratch just became 16MB on distroless. That 4MB overhead buys you the ability to run openssl checks and properly resolve DNS in Kubernetes.

Here's the decision tree: If your app needs SSL certificates (HTTP clients, database connections, gRPC), use distroless. If your app runs as PID 1 and never talks to the outside world, scratch is fine. But if you ever need to exec into a container and troubleshoot — distroless gives you just enough rope without the noose.

The real hack? Start with distroless, then strip back to scratch only when you've profiled exactly what's unused. Most teams overestimate their binary's independence and underestimate debugging time.

// io.thecodeforge — devops tutorial

FROM golang:1.21 AS build
WORKDIR /app
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /app/payment-processor .

# Option A: Scratch (minimal)
FROM scratch AS scratch-production
COPY --from=build /app/payment-processor /payment-processor
CMD ["/payment-processor"]

# Option B: Distroless (safe)
FROM gcr.io/distroless/base-debian12 AS distroless-production
COPY --from=build /app/payment-processor /payment-processor
CMD ["/payment-processor"]

# Distroless image size: 16MB vs scratch: 12MB
# But distroless has SSL certs, ca-certificates, tzdata, no shell

The Chunky Single-Stage Build

A single-stage Dockerfile installs everything—compiler, test tools, dev dependencies, and runtime—into one image. Each RUN command adds a new layer, but even if you delete files in the same layer, the file data remains in the previous layer, bloating the final image. The real problem: your production container carries a compiler and build artifacts it never executes. This violates the principle of least privilege and expands the attack surface. A chunky build also slows CI, because every layer change invalidates downstream caches. When you push a minor code fix, Docker rebuilds layers from the first change, not from the optimized base. The fix isn't more rm commands—it's separating build and runtime into distinct stages. Single-stage builds feel simple but cost you security, speed, and image hygiene.

// io.theforge — devops tutorial

// Chunky single-stage: compiler + runtime in one image
FROM golang:1.21 AS builder
WORKDIR /app
COPY . .
RUN go build -o app .

# Bloat: contains compiler, test tools, source code
ENTRYPOINT ["./app"]

How to Implement Multi-Stage Builds

Multi-stage builds let you use multiple FROM statements in one Dockerfile. Each FROM begins a new stage, and only files you explicitly COPY from previous stages survive to the final image. The pattern: build stage installs dependencies and compiles; runtime stage starts from a minimal base (e.g., alpine, distroless) and copies only the binary. This drops image size by 80-90% and removes tooling that attackers could exploit. You can name stages—builder, test, production—and target them with --target. Implementation requires no extra tooling; Docker's built-in parser handles it. Start by moving all build steps (go build, npm install, mvn package) into a stage named build. Then create a production stage with COPY --from=build. Validate with docker history to see only runtime layers survive.

// io.theforge — devops tutorial

// Multi-stage: minimal final image
FROM golang:1.21 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -o bin/app .

FROM alpine:3.19
RUN apk add --no-cache ca-certificates
COPY --from=build /src/bin/app /app
ENTRYPOINT ["/app"]