Advanced 5 min · March 06, 2026

Multi-Stage Docker Builds — Secrets Survive rm

Q: What is a multi-stage Docker build?

A Dockerfile with multiple FROM instructions, each starting an isolated build stage. The build stage contains compilers and tools. The final stage contains only the runtime artifacts copied from the build stage via COPY --from. Build tools never ship in the final image.

Q: How much smaller are multi-stage images?

Typically 90-97% smaller. A Go application goes from ~800 MB (full golang base) to ~20 MB (scratch with static binary). A Node.js application goes from ~1.2 GB (full node base with devDependencies) to ~150 MB (alpine with production dependencies only).

Q: Do I need BuildKit for multi-stage builds?

Multi-stage builds work without BuildKit, but you lose cache mounts, secret mounts, and parallel stage execution. BuildKit is required for the security and performance features that make multi-stage builds production-grade. Always enable it.

Q: How do I debug a container built from a minimal image?

Use ephemeral debug containers: `kubectl debug -it --image=busybox --target= `. Or add a debug stage that includes debugging tools and use `docker build --target debug` when you need it. Never ship debugging tools in the production image.

Q: Can I use multi-stage builds with docker-compose?

Yes. docker-compose supports multi-stage Dockerfiles natively. You can also specify a target stage with `target: runtime` in the build section to stop at a specific stage (useful for development builds that need the full toolchain).

Even after 'rm', secrets persist in Docker layers—token was recovered by inspecting history.

Naren · Founder

Plain-English first. Then code. Then the interview question.

About

● Production Incident 🔎 Debug Guide

⚡Quick Answer

Core mechanism: Each FROM starts a new stage. Only files explicitly copied via COPY --from= end up in the final image. Everything else — compilers, build caches, source code — is discarded.
Size impact: Typical reduction from 1.2 GB (single-stage) to 80-150 MB (multi-stage). The build tools never ship.
Layer caching: BuildKit caches each stage independently. Changing application code does not invalidate the dependency-install stage.
BuildKit parallelism: Stages with no dependency between them execute in parallel, cutting CI time.
Security: Build secrets (API keys, tokens) used in early stages never appear in the final image layers.
Biggest mistake: Forgetting that only explicitly COPYed artifacts survive. If you build a binary in stage 1 but forget to COPY it in stage 2, the final image has nothing to run.

Plain-English First

Imagine you're baking a cake. You need mixing bowls, electric beaters, and measuring cups to make it — but when you serve the cake to guests, you don't put all that equipment on the plate with it. Multi-stage Docker builds work the same way: one stage is your messy kitchen where all the building happens, and the final stage is just the clean, finished cake. Your users get the cake. The mixing bowls stay in the kitchen — and never ship to production.

Every second your Docker image takes to pull across a network is a second your deployment is stalled. In Kubernetes environments rolling out hundreds of pods under load, or CI pipelines building dozens of images a day, bloated images are a reliability and cost problem. A Node.js app shipping with its full devDependencies, TypeScript compiler, and build toolchain alongside the production binary is a 1.2 GB image waiting to become a 3 AM outage.

Traditional single-stage Dockerfiles are all-or-nothing. You install the compiler, build the binary, copy the source — and all of it ends up baked into the final layer. Docker does not have a native concept of 'clean up after yourself' within a single build context, because every RUN instruction adds a new immutable layer. Removing files in a later layer does not reclaim space — it just hides them.

Multi-stage builds solve this by introducing multiple isolated build contexts inside one Dockerfile. Each FROM starts a fresh stage with its own filesystem. Only artifacts you explicitly copy forward survive into the final image. The build tools, intermediate object files, and source code are discarded with the build stage. This is the single most impactful Dockerfile optimization for production images.

How Multi-Stage Builds Work at the Layer Level

A Dockerfile with multiple FROM instructions creates multiple isolated stages. Each stage starts with a fresh filesystem initialised from its base image. Stages are identified by their index (0, 1, 2...) or by an alias assigned with AS.

The critical insight: only files you explicitly COPY --from=<stage> are transferred between stages. Everything else — compilers, build caches, intermediate object files, source code — exists only in the build stage's filesystem and is discarded when the build completes. The final image contains only the last stage's filesystem plus any files copied into it.

Docker's layer system means each RUN, COPY, and ADD instruction creates an immutable layer. In a single-stage build, RUN rm file creates a NEW layer that hides the file — the original layer with the file still exists in the image. Multi-stage builds avoid this entirely by never including the file in the final stage's layers in the first place.

Dockerfile.go-multistageDOCKERFILE

# ─────────────────────────────────────────────────────────────
# Stage 1: Build stage — contains Go compiler, source, dependencies
# This stage is ~800 MB but NEVER ships to production
# ─────────────────────────────────────────────────────────────
FROM golang:1.22-alpine AS builder

WORKDIR /app

# Copy dependency files FIRST — this layer is cached until go.mod changes
COPY go.mod go.sum ./
RUN go mod download

# Copy source code AFTER dependencies — changing source doesn't invalidate dep cache
COPY . .

# Build a statically linked binary — no runtime dependencies needed
# CGO_ENABLED=0 ensures no C library dependency
# -ldflags='-s -w' strips debug symbols, reducing binary size by ~30%
RUN CGO_ENABLED=0 GOOS=linux go build \
    -ldflags='-s -w' \
    -o /app/server \
    ./cmd/server

# ─────────────────────────────────────────────────────────────
# Stage 2: Runtime stage — contains ONLY the binary and config
# This stage is ~15-25 MB — a 97% reduction from the build stage
# ─────────────────────────────────────────────────────────────
FROM alpine:3.19 AS runtime

# Install CA certificates for HTTPS and tzdata for timezone support
RUN apk --no-cache add ca-certificates tzdata

WORKDIR /app

# Copy ONLY the compiled binary from the builder stage
# Everything else (Go compiler, source, build cache) is discarded
COPY --from=builder /app/server .

# Copy config files if needed
COPY --from=builder /app/config ./config

# Run as non-root user — security best practice
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

EXPOSE 8080

ENTRYPOINT ["./server"]

Output

# Build command:

# DOCKER_BUILDKIT=1 docker build -t myapp:latest .

# Size comparison:

# golang:1.22-alpine base: ~800 MB

# Builder stage with binary: ~820 MB

# Final runtime stage: ~22 MB (97% reduction)

# Verify with:

# docker images myapp:latest

# REPOSITORY TAG SIZE

# myapp latest 22.4MB

Stages Are Isolated Filesystems, Not Sequential Steps

Each FROM creates a new isolated filesystem
Data moves between stages ONLY via COPY --from
Build stage is destroyed after build — its layers never ship
Final image = last stage filesystem only
Secrets in early stages cannot leak into final stage unless explicitly copied

Production Insight

Cause: Developers copy the entire project directory (COPY . .) before installing dependencies. When any source file changes, Docker invalidates the COPY layer and every subsequent layer — including the dependency installation layer. Effect: Every code change triggers a full npm install or go mod download, adding 30-120 seconds to build time depending on dependency count. In CI, this multiplies across dozens of daily builds. Action: Copy dependency manifests (package.json, go.mod, pom.xml) BEFORE copying source code. Install dependencies in a layer that only invalidates when the manifest changes. This single reordering typically cuts rebuild time by 60-80%.

Key Takeaway

Multi-stage builds separate build tools from runtime artifacts. Only explicitly COPYed files survive. The final image contains only the last stage. This is not a cleanup strategy — it is a fundamentally different build model where the build environment never ships.

Choosing the Final Stage Base Image

IfCompiled binary (Go, Rust) with no runtime dependencies.

→

UseUse FROM scratch (~0 MB) or FROM alpine (~7 MB) with ca-certificates for HTTPS.

IfNode.js application requiring native modules.

→

UseUse FROM node:20-slim (~200 MB). Alpine may fail on native modules that require glibc.

IfJava application requiring JVM.

→

UseUse FROM eclipse-temurin:21-jre-alpine (~100 MB). JRE only, not JDK — the compiler is not needed at runtime.

IfPython application with C extensions (numpy, pandas).

→

UseUse FROM python:3.12-slim (~150 MB). Alpine's musl libc fails on C extensions.

IfMaximum security — zero shell, zero package manager in final image.

→

UseUse FROM scratch with a statically linked binary. No shell, no attacks — but no debug tools either.

Stage Targeting with --target: Production vs Build vs Test

Docker allows you to stop the build at any named stage using the --target flag. This is invaluable for development workflows, CI, and debugging. Without --target, Docker builds all stages up to the last FROM. With --target, you can request only the stages you need.

Common use cases

Development: Build only up to the deps or builder stage, which includes all dev dependencies and tooling. Developers get a container with live-reload tools (e.g., nodemon, reflex) without waiting for the full production image to build.
Testing: Build a test stage that runs unit tests, linters, and security scans. CI can pull this stage, run tests, and discard it without ever building the final production image.
Production: Build the final runtime stage (runtime or production) that contains only what ships to production.

How to name stages and target them

Use AS <name> on every FROM you may want to target.
The last FROM that isn't used as a base for other stages is the default target.
Build command: docker build --target production -t myapp:latest .

Dockerfile.target-exampleDOCKERFILE

# ─────────────────────────────────────────────────────────────
# Multi-stage Dockerfile with explicit stage targets
# Build with: docker build --target <stage> -t image .
# ─────────────────────────────────────────────────────────────

# Stage 0: Base dependencies (shared across build and dev)
FROM node:20-alpine AS base
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci

# Stage 1: Development — includes dev dependencies and tools
FROM base AS development
RUN npm ci --include=dev
COPY . .
RUN npm install -g nodemon
CMD ["nodemon", "src/index.js"]

# Stage 2: Test — runs tests with coverage
FROM base AS test
RUN npm ci --include=dev
COPY . .
RUN npm test

# Stage 3: Build — compiles TypeScript, builds assets
FROM base AS build
COPY . .
RUN npm run build

# Stage 4: Production — minimal runtime image
FROM node:20-alpine AS production
WORKDIR /app
# Copy only production dependencies from base
COPY --from=base /app/node_modules ./node_modules
# Copy built application from build stage
COPY --from=build /app/dist ./dist
COPY --from=build /app/package.json ./
USER node
CMD ["node", "dist/index.js"]

Output

# Build commands:

# Development: docker build --target development -t myapp:dev .

# Test (no final image): docker build --target test -t myapp:test .

# Production: docker build --target production -t myapp:prod .

# Use --target with docker-compose:

# build:

# context: .

# target: production

--target Does Not Prune Previous Stages

--target includes the named stage and all its upstream dependencies
Stages not in the dependency chain are skipped completely
Use --target in CI to build test image without waiting for production build
Great for parallel CI: test stage and production stage can be built independently

Production Insight

Cause: CI builds the entire multi-stage Dockerfile every time, even when only tests need to run. Effect: Test runs wait for the full production image to build (including compilation and asset minification) before any test execution begins. Action: Use docker build --target test to build only the test stage. Run tests in that image, then docker build --target production only on successful test passes. This cuts CI pipeline time by 30-50% for typical workflows.

Key Takeaway

docker build --target <stage> gives you surgical control over which stages execute. Use it to create efficient CI pipelines that separate test from production builds, and to give developers fast feedback images. Name every stage you might want to target.

When to Use Each Target

IfLocal development — need hot-reload and debug tools.

→

UseUse --target development or --target dev. This stage has full toolchain and dev dependencies.

IfCI test run — no need for production image.

→

UseUse --target test. Build only dependencies + test execution stage. Faster feedback.

IfProduction deployment.

→

UseUse --target production (or the last stage, which is the default). Minimal, secure, no build tools.

IfDebugging build failures — inspect build stage without full build.

→

UseUse --target builder or the stage that fails. Quickly inspect state with docker run -it <stage-image> /bin/sh.

BuildKit, Parallel Stages, and Cache Mounts

BuildKit is Docker's modern build engine, enabled by setting DOCKER_BUILDKIT=1 or using docker buildx. It brings three critical capabilities to multi-stage builds:

Parallel stage execution: Stages that do not depend on each other run concurrently. If your Dockerfile has a test stage and a build stage that both depend on the dependency-install stage, BuildKit runs test and build in parallel after dependencies are installed.
Cache mounts: --mount=type=cache persists a directory across builds without invalidating the layer. This is transformative for package managers — mount the npm/pip/go cache directory so dependency downloads are cached across builds even when the layer would otherwise be invalidated.
Secret mounts: --mount=type=secret provides a temporary file during RUN execution that is never stored in any layer. This is the correct way to use API keys, tokens, and credentials during builds.

Dockerfile.buildkit-advancedDOCKERFILE

# syntax=docker/dockerfile:1  # required for BuildKit features

# ─────────────────────────────────────────────────────────────
# Stage 1: Dependency installation with cache mount
# The npm cache persists across builds — re-installs are near-instant
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS deps

WORKDIR /app

COPY package.json package-lock.json ./

# --mount=type=cache persists /root/.npm across builds
# Even if this layer is invalidated, the cache is not lost
RUN --mount=type=cache,target=/root/.npm \
    npm ci --omit=dev

# ─────────────────────────────────────────────────────────────
# Stage 2: Build stage with secret mount for private registry
# The npm token is available during build but never stored in a layer
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS builder

WORKDIR /app

COPY package.json package-lock.json ./

# Secret mount: token is available as a file during this RUN only
# It is NEVER stored in any layer — not even in a hidden layer
RUN --mount=type=secret,id=npm_token,target=/run/secrets/npm_token \
    --mount=type=cache,target=/root/.npm \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

COPY . .

RUN npm run build

# ─────────────────────────────────────────────────────────────
# Stage 3: Production runtime — minimal image
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS runtime

WORKDIR /app

# Copy only production dependencies from deps stage
COPY --from=deps /app/node_modules ./node_modules

# Copy built application from builder stage
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./

# Run as non-root
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
USER appuser

EXPOSE 3000

CMD ["node", "dist/index.js"]

# ─────────────────────────────────────────────────────────────
# Build command with secret:
# DOCKER_BUILDKIT=1 docker build \
#   --secret id=npm_token,src=.npmrc_token \
#   -t myapp:latest .

Output

# Build output showing parallel stage execution:

# [+] Building 12.3s (14/14) FINISHED

# => [deps 2/2] RUN --mount=type=cache npm ci 8.2s (cached)

# => [builder 3/4] RUN --mount=type=secret npm ci 9.1s

# => [builder 4/4] RUN npm run build 3.2s

# => [runtime 3/3] COPY --from=builder /app/dist 0.1s

# Second build (only source code changed):

# [+] Building 4.1s (14/14) FINISHED ← deps cached, only build stage reruns

# => [deps 2/2] RUN --mount=type=cache npm ci 0.3s (cache hit)

# => [builder 3/4] RUN npm ci 0.5s (cache hit)

# => [builder 4/4] RUN npm run build 3.2s (only this reruns)

Watch Out: --mount=type=cache Is Not Layer Caching

Layer cache: if inputs unchanged, skip layer entirely (0s)
Cache mount: if layer re-runs, reuse downloaded packages (fast re-download)
Use both together for maximum build speed
Cache mounts require BuildKit — they do not work with the legacy builder

Production Insight

Cause: CI pipelines rebuild Docker images on every commit. Without cache mounts, every build downloads all dependencies from scratch — even if only one source file changed. With 200+ npm packages, this adds 60-90 seconds per build. Effect: CI pipeline duration becomes dominated by dependency download time, not actual build time. Developer feedback loop slows, and CI costs increase proportionally. Action: Add --mount=type=cache for package manager caches. Combine with proper layer ordering (dependency manifest before source code). This typically cuts CI build time from 3-5 minutes to 30-60 seconds for incremental changes.

Key Takeaway

BuildKit transforms multi-stage builds from a size optimisation into a full build acceleration platform. Cache mounts persist package downloads across builds. Secret mounts keep credentials out of layers entirely. Parallel stage execution cuts CI time. All three require BuildKit — the legacy builder supports none of them.

When to Use Each BuildKit Feature

IfNeed to use API keys, tokens, or credentials during build.

→

UseUse --mount=type=secret. Available as a file during RUN, never stored in any layer.

IfPackage manager downloads are slow on every build.

→

UseUse --mount=type=cache targeting the package manager cache (e.g., /root/.npm, /root/.cache/pip).

IfBuild has independent stages (test + build from same deps).

→

UseEnsure BuildKit is enabled. It automatically parallelises independent stages.

IfNeed to share a file between stages without copying it into the final image.

→

UseUse an intermediate stage. COPY the file in, then COPY --from=intermediate in the final stage.

Production Patterns: Go, Node.js, and Java

Each language ecosystem has specific multi-stage build patterns that address its unique characteristics. The patterns below are battle-tested in production and handle the most common failure modes.

Dockerfile.java-multistageDOCKERFILE

# ─────────────────────────────────────────────────────────────
# Java Multi-Stage Build: Maven build + JRE runtime
# Reduces image from ~800 MB (JDK) to ~120 MB (JRE alpine)
# ─────────────────────────────────────────────────────────────

# Stage 1: Build with Maven
FROM maven:3.9-eclipse-temurin-21 AS builder

WORKDIR /app

# Copy POM first — this layer is cached until pom.xml changes
COPY pom.xml .

# Download dependencies with cache mount
RUN --mount=type=cache,target=/root/.m2 \
    mvn dependency:go-offline -B

# Copy source and build
COPY src ./src
RUN --mount=type=cache,target=/root/.m2 \
    mvn package -DskipTests -B

# Stage 2: Extract JRE runtime
FROM eclipse-temurin:21-jre-alpine AS runtime

WORKDIR /app

# Copy only the fat JAR from the build stage
COPY --from=builder /app/target/*.jar app.jar

# JVM tuning for containers — critical for production
# -XX:+UseContainerSupport respects cgroup memory limits
# -XX:MaxRAMPercentage=75% uses 75% of container memory for heap
ENV JAVA_OPTS="-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0 -XX:+UseG1GC"

EXPOSE 8080

ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

Output

# Size comparison:

# maven:3.9-eclipse-temurin-21: ~800 MB (build stage — discarded)

# eclipse-temurin:21-jre-alpine: ~120 MB (runtime stage — ships)

# Final image with app JAR: ~145 MB

Production Pattern: Separate Dependency Cache from Build

Copy dependency manifest (pom.xml, go.mod, package.json) FIRST
Install/download dependencies in a separate layer
Copy source code AFTER dependencies are installed
Build/compile in the final layer
Changing source code only invalidates the build layer, not the dependency layer

Production Insight

Cause: Using the full JDK as the runtime base image. The JDK includes the compiler (javac), debugger (jdb), and development tools that are never used at runtime. Effect: The runtime image is 600-800 MB instead of 100-150 MB. In Kubernetes environments with 50+ pods, this means 30-40 GB of unnecessary image pulls per deployment rollout, adding minutes to rollout time and significant registry egress costs. Action: Always use JRE (not JDK) for the runtime stage. Use alpine variants where possible. For maximum minimalism, use jlink to create a custom JRE containing only the modules your application needs.

Key Takeaway

Every language ecosystem has a specific multi-stage pattern. The universal rule: copy dependency manifests before source code, use the smallest possible runtime base, and never ship the compiler to production. The specific base image choice depends on whether your runtime needs glibc or musl, and whether native extensions are required.

Language-Specific Runtime Base Image Selection

IfGo application — statically linked binary.

→

UseUse FROM scratch (0 MB). Binary runs directly. COPY ca-certificates from alpine for HTTPS.

IfNode.js application — no native modules.

→

UseUse FROM node:20-alpine (~120 MB). Fastest pull, smallest footprint.

IfNode.js application — requires native modules (bcrypt, sharp).

→

UseUse FROM node:20-slim (~200 MB). Alpine's musl libc breaks some native modules.

IfJava application — standard Spring Boot.

→

UseUse FROM eclipse-temurin:21-jre-alpine (~120 MB). JRE only, not JDK.

IfPython application — pure Python dependencies.

→

UseUse FROM python:3.12-alpine (~50 MB). Minimal footprint.

IfPython application — numpy, pandas, scipy (C extensions).

→

UseUse FROM python:3.12-slim (~150 MB). Alpine requires recompiling C extensions.

Image Size Reduction Comparison Table

One of the strongest arguments for multi-stage builds is the dramatic reduction in image size. Below is a comparison of typical image sizes for common language stacks using single-stage vs. optimal multi-stage builds. These numbers are based on production images (source code + dependencies + runtime) and assume best practices like using slim/alpine bases and stripping debug symbols.

The pattern is consistent: build tools and compilers account for 70-90% of image bulk. Multi-stage builds eliminate them from the final image, leaving only the runtime and the compiled artifacts.

Size Depends on Runtime Requirements

Go and Rust can use FROM scratch – often < 20 MB
Node.js minimum ~120 MB (node:20-alpine), but includes Node runtime
Java JRE alpine ~120 MB, but JDK adds ~700 MB
Python slim ~150 MB, but C extensions add 200-400 MB

Production Insight

Cause: Teams migrate to multi-stage but keep using full-size base images for the runtime stage (e.g., node:20 instead of node:20-alpine). Effect: The image is still 900 MB instead of 150 MB, defeating the purpose of multi-stage. Action: Always pair multi-stage with the smallest possible runtime base. For Node.js, use alpine unless native modules require glibc. For Java, use JRE not JDK. For Go, use scratch if no OS dependencies.

Key Takeaway

Multi-stage builds consistently reduce image size by 80-98%. The final size is bounded by the runtime base image. Choose the smallest base that supports your application's runtime requirements. Statically compiled binaries give the smallest images.

Multi-Stage Build Example: Go (Minimal Pattern)

This section provides a focused, minimal multi-stage Dockerfile for a Go application. It demonstrates the canonical pattern: build stage containing the compiler and source, and a runtime stage with only the binary. The result is a 97% reduction in image size.

Key steps: 1. Use golang:alpine as the build stage – includes Go compiler and standard library, but is smaller than full debian-based images. 2. Copy go.mod/go.sum first – layer caching ensures go mod download runs only when dependencies change. 3. Build with CGO_ENABLED=0 – produces a statically linked binary that does not depend on glibc, allowing FROM scratch as runtime. 4. Use -ldflags='-s -w' – strips debug symbol tables, reducing binary size by ~30%. 5. Runtime base: scratch – zero bytes, no shell, no package manager. Maximum security and minimal size.

Dockerfile.go-minimalDOCKERFILE

# syntax=docker/dockerfile:1
# Minimal Go multi-stage build

# Stage 1: Build the Go binary
FROM golang:1.22-alpine AS builder

WORKDIR /build

# Download dependencies first - leverages layer caching
COPY go.mod go.sum ./
RUN go mod download

# Copy source code
COPY . .

# Build a static binary, strip debug symbols
RUN CGO_ENABLED=0 GOOS=linux go build \
    -ldflags='-s -w' \
    -o /build/server \
    ./cmd/server

# Stage 2: Runtime - absolutely minimal
FROM scratch AS runtime

# Copy CA certificates for HTTPS calls (from builder stage)
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/

# Copy the binary
COPY --from=builder /build/server /server

# No shell, no package manager, no user (not needed - scratch has /dev/null etc.)
EXPOSE 8080

ENTRYPOINT ["/server"]

Output

# Sizes:

# golang:1.22-alpine: ~800 MB

# builder stage: ~820 MB (temporary)

# runtime stage (scratch): ~18 MB (binary + CA certs only)

# Reduction: 97.8%

# Build command:

# DOCKER_BUILDKIT=1 docker build -t go-app:latest .

Why Use scratch Instead of alpine for Go?

CGO_ENABLED=0 produces a fully static binary
scratch has no shell, no utilities – secure but limited
Add ca-certificates from builder for HTTPS calls
For debugging, use ephemeral debug containers

Production Insight

Cause: Teams using multi-stage for Go applications sometimes forget to set CGO_ENABLED=0 and copy the binary into alpine instead of scratch. Effect: The binary becomes dynamically linked against glibc, requiring the full glibc library in the runtime image – adding 5-10 MB but more importantly, making the image depend on the host OS's libc. Action: Always set CGO_ENABLED=0 in Go builds targeting scratch. Verify binary is static with file server (should say 'statically linked').

Key Takeaway

For Go applications, the canonical multi-stage pattern uses golang:alpine as the build stage and scratch as the runtime stage. The result is an 18 MB image that contains exactly one binary. No shell, no compiler, no attack surface.

Go Runtime Base: scratch vs alpine vs distroless

IfBinary built with CGO_ENABLED=0, no OS dependencies.

→

UseUse FROM scratch. Smallest and most secure. Downside: no shell for debugging.

IfBinary requires glibc (CGO_ENABLED=1) or you need common tools like curl.

→

UseUse FROM alpine:3.19 (5-7 MB) and install only what you need. Avoid debian-based images unless necessary.

IfNeed a minimal image with common utilities but no package manager.

→

UseUse FROM gcr.io/distroless/base. Includes glibc, tzdata, but no shell or package manager. Good balance.

Security: Secret Management and Image Scanning

Multi-stage builds are a security primitive, not just a size optimisation. The build stage isolation means secrets used during compilation never appear in the final image — IF you use the correct mechanisms. The wrong mechanism (ENV, ARG, or inline RUN) leaks secrets into layers permanently.

Three rules for secret management in Docker builds: 1. Never use ENV or ARG for secrets — they persist in image metadata. 2. Never hardcode secrets in RUN commands — they persist in layer history. 3. Always use BuildKit secret mounts — --mount=type=secret provides temporary file access without layer persistence.

Beyond secrets, the final image should be scanned for vulnerabilities. Even a minimal alpine base image may contain packages with known CVEs. Trivy, Grype, and Snyk Container can scan images in CI and block deployment if critical vulnerabilities are found.

Dockerfile.secure-patternDOCKERFILE

# syntax=docker/dockerfile:1

# ─────────────────────────────────────────────────────────────
# WRONG: Secrets in ENV or ARG — visible in docker history and inspect
# ─────────────────────────────────────────────────────────────
# DO NOT DO THIS:
# ENV NPM_TOKEN=ghp_xxxxxxxxxxxx           ← visible in docker inspect
# ARG DB_PASSWORD=secret123                ← visible in docker history
# RUN echo $NPM_TOKEN > .npmrc             ← visible in layer history

# ─────────────────────────────────────────────────────────────
# CORRECT: BuildKit secret mounts
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS builder

WORKDIR /app

COPY package.json package-lock.json ./

# Secret is mounted as a file at /run/secrets/npm_token
# Available only during this RUN command
# Not stored in any layer — not even hidden
RUN --mount=type=secret,id=npm_token,target=/run/secrets/npm_token \
    --mount=type=cache,target=/root/.npm \
    NPM_TOKEN=$(cat /run/secrets/npm_token) npm ci

COPY . .
RUN npm run build

# ─────────────────────────────────────────────────────────────
# Final stage: no secrets, no build tools, no source code
# ─────────────────────────────────────────────────────────────
FROM node:20-alpine AS runtime

WORKDIR /app

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/package.json ./

USER node

CMD ["node", "dist/index.js"]

# ─────────────────────────────────────────────────────────────
# Build command:
# DOCKER_BUILDKIT=1 docker build \
#   --secret id=npm_token,src=./.npm_token_value \
#   -t myapp:latest .
#
# Verify no secrets in image:
# docker history myapp:latest | grep -i token  # should return nothing
# docker inspect myapp:latest | grep -i token   # should return nothing

Output

# Verification:

# $ docker history myapp:latest

# IMAGE CREATED BY SIZE

# abc123 CMD ["node" "dist/index.js"] 0B

# def456 COPY /app/dist ./dist # buildkit 2.1MB

# ...

# No secret tokens visible anywhere in the history

Watch Out: ARG Values Persist in Image Metadata

ARG values appear in docker inspect metadata
ENV values appear in docker inspect and are available to all subsequent layers
RUN echo secret > file stores the secret in that layer permanently
--mount=type=secret is the ONLY mechanism that does not persist the secret

Production Insight

Cause: Using ENV or ARG to pass secrets to Docker builds. These values persist in image metadata and layer history, making them extractable by anyone with pull access to the image. Effect: Credentials leak to anyone who can run docker inspect or docker history on the image. In shared registries, this means every developer and every CI system with pull access can extract production secrets. Action: Use BuildKit secret mounts exclusively. Add Trivy or Grype scanning to CI to detect accidentally embedded secrets. Run docker history <image> as a post-build verification step.

Key Takeaway

Multi-stage builds are a security boundary. Secrets used in early stages never appear in the final image — but only if you use BuildKit secret mounts. ENV, ARG, and inline RUN commands all leak secrets into layers. Always scan images in CI before pushing to any registry.

Secret Handling: Correct vs Incorrect Patterns

IfNeed an API key during npm install or pip install.

→

UseUse --mount=type=secret,id=token to mount the secret as a file during RUN. Never stored in any layer.

IfNeed a value that is NOT secret but varies per build (version number, build target).

→

UseUse ARG. This is the correct use case — non-secret build-time variables.

IfNeed a value available at runtime (not build time).

→

UsePass via docker run -e VAR=value or Kubernetes env vars — not in the Dockerfile.

IfNeed to verify no secrets are in the final image.

→

UseRun docker history <image> then docker inspect <image>. Search for leaked tokens. Add Trivy to CI.

● Production incidentPOST-MORTEMseverity: high

Secrets Leaked in Docker Image Layers Exposed API Keys to Public Registry

Symptom

Security audit flagged that the company's private npm packages were being downloaded by unknown external IPs. An attacker had extracted the npm token from a layer in the public Docker image and used it to access the company's private registry.

Assumption

The team assumed that RUN rm /root/.npmrc after npm install would remove the token from the image. They did not understand that Docker layers are immutable — the rm creates a new layer that hides the file, but the original layer containing the token is still present in the image history.

Root cause

The Dockerfile used a single-stage build. The npm token was written to /root/.npmrc in one RUN layer, npm install ran in the next layer, and RUN rm /root/.npmrc ran in a third layer. The rm command only marked the file as deleted in the new layer — the token was still recoverable from the earlier layer by inspecting docker history or extracting layers manually. The image was pushed to a public ECR registry without secret scanning.

Fix

1. Converted to a multi-stage build. The build stage uses --mount=type=secret to mount the npm token as a temporary file that never persists in any layer. 2. Added DOCKER_BUILDKIT=1 to CI to enable BuildKit secret mounts. 3. Added Trivy and Hadolint scanning to CI pipeline — both detect secrets in image layers. 4. Rotated the compromised npm token immediately. 5. Made the registry private and added IP-based access controls.

Key lesson

Docker layers are immutable. RUN rm does not remove data — it creates a new layer that hides the file. The original data is still in the image.
Never embed secrets in RUN commands. Use BuildKit secret mounts (--mount=type=secret) or multi-stage builds where the secret stage is discarded.
Always run image security scanning (Trivy, Grype, Snyk Container) in CI before pushing to any registry.
Public registries are hostile environments. Assume every layer will be inspected by an attacker.

Production debug guideWhen the final image is missing files, too large, or builds are slower than expected.5 entries

Symptom · 01

Final image is missing the binary or application files — container exits immediately with 'file not found'.

→

Fix

Verify that the COPY --from=builder path matches the actual build output location. Common mistake: building with WORKDIR /app in the build stage but copying from /build/output in the final stage. Check docker run --rm -it <build-stage-image> ls /app to see what the build stage actually produced.

Symptom · 02

Final image is still 1+ GB despite using multi-stage build.

→

Fix

Check the base image of the final stage. Using FROM node:20 as the runtime base pulls in the full Node.js SDK (~900 MB). Switch to FROM node:20-slim (~200 MB) or FROM node:20-alpine (~120 MB). Also verify that devDependencies are not installed in the final stage — use npm ci --omit=dev.

Symptom · 03

Build is slower with multi-stage than single-stage — expected the opposite.

→

Fix

Check if BuildKit is enabled. Without BuildKit, Docker builds stages sequentially even when they could run in parallel. Set DOCKER_BUILDKIT=1 or add "features": {"buildkit": true} to /etc/docker/daemon.json. Also check if layer caching is being invalidated by copying files too early in the Dockerfile.

Symptom · 04

Build cache is invalidated on every build even when only application code changed.

→

Fix

The COPY instruction for source code is before the dependency install. Docker invalidates all layers after a changed COPY. Move dependency installation (package.json, go.mod, pom.xml) to before the source code COPY so the dependency cache survives code changes.

Symptom · 05

Secrets visible in docker history <image> output.

→

Fix

The secret was used in a RUN command without BuildKit secret mounts. Convert to RUN --mount=type=secret,id=npm_token npm install and pass the secret at build time with --secret id=npm_token,src=.npmrc. Verify with docker history that the secret does not appear.

★ Docker Multi-Stage Build Triage Cheat SheetFast diagnostics for image size, build speed, and missing artifact issues.

Final image is unexpectedly large (>500 MB for a compiled binary).−

Immediate action

Check final stage base image and installed packages.

Commands

docker history <image> --no-trunc # inspect layer sizes

docker run --rm <image> du -sh /* # find large directories

Fix now

Switch to alpine/slim base. Remove devDependencies. Verify only runtime artifacts are COPYed.

Binary or app files missing in final image.+

Build cache invalidated on every run.+

Secrets visible in docker history.+

Multi-stage build slower than expected — stages not running in parallel.+

Single-Stage vs Multi-Stage Docker Builds

Aspect	Single-Stage Build	Multi-Stage Build
Dockerfile structure	One FROM instruction	Multiple FROM instructions, each starting a new stage
Final image contents	Everything: compilers, source, build tools, runtime	Only runtime artifacts explicitly copied from build stages
Typical image size	800 MB – 1.5 GB	20 MB – 200 MB (90%+ reduction)
Secret handling	Secrets persist in layers unless manually removed (unsafe)	Secrets in build stages never appear in final image
Layer caching	Single cache chain — any change invalidates downstream layers	Per-stage caching — dependency stage cached independently of build stage
BuildKit parallelism	Not applicable — single sequential build	Independent stages execute in parallel
Security surface	Large — shell, package manager, compilers all present	Minimal — only runtime dependencies, often no shell
Debugging in production	Easy — full toolchain available in container	Harder — use ephemeral debug containers or distroless+debug images
CI build time	Slow — full rebuild on any code change	Fast — dependency layer cached, only build layer reruns

Key takeaways

Multi-stage builds separate build tools from runtime artifacts. Only explicitly COPYed files survive. The final image contains only the last stage.

BuildKit is not optional

cache mounts, secret mounts, and parallel stage execution all require it. Enable it in every CI pipeline.

Dependency manifest before source code. Always. This single reordering cuts rebuild time by 60-80%.

Secrets in ENV, ARG, or inline RUN commands persist in image layers. Use --mount=type=secret exclusively.

The runtime base image determines your security surface and pull time. Choose the smallest base that supports your runtime requirements.

RUN rm does not remove data from images. Multi-stage builds are the correct mechanism for excluding build artifacts.

INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

FAQ · 5 QUESTIONS

Frequently Asked Questions

What is a multi-stage Docker build?

How much smaller are multi-stage images?

Do I need BuildKit for multi-stage builds?

How do I debug a container built from a minimal image?

Can I use multi-stage builds with docker-compose?

🔥

That's Docker. Mark it forged?

5 min read · try the examples if you haven't