Docker with GitLab CI/CD: Build Pipelines That Don't Burn You at 3 AM

You've seen the email: 'Pipeline failed — no space left on device.' Or worse, the 3 AM page because a Docker build that worked locally pushed a broken image to production. Docker with GitLab CI/CD is the backbone of modern DevOps, but most teams treat it like a black box. They copy-paste a .gitlab-ci.yml from a blog, cross their fingers, and hope. That works until it doesn't — and when it breaks, it breaks hard. This article gives you the battle-tested patterns to build pipelines that don't burn you. You'll learn how to cache like a pro, avoid the Docker-in-Docker trap, and debug the failures that make juniors cry. By the end, you'll be able to design a pipeline that's fast, reliable, and actually safe for production.

Why Your Pipeline Needs Docker (And Why It Doesn't)

Docker gives you reproducibility. Your build runs in the exact same environment every time — same OS packages, same Node version, same glibc. Without it, you get the classic 'works on my machine' nightmare. But Docker isn't free. It adds complexity: you need a Docker daemon in your CI, you need to manage image caches, and you need to handle layer bloat. If your app is a simple static site that just needs npm install && npm build, you might be better off using a pre-built image with the tools you need, skipping Docker entirely. The rule: use Docker when you need to ship a container. Use a plain shell executor when you just need to run tests or lint.

// io.thecodeforge — DevOps tutorial

# Minimal Docker pipeline for a Node.js app
image: node:18-alpine

services:
  - docker:dind

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_DRIVER: overlay2
  DOCKER_TLS_CERTDIR: ""

before_script:
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

Layer Caching: The Art of Not Rebuilding the World

Every RUN command in your Dockerfile creates a layer. If you change a file early in the Dockerfile, all subsequent layers are invalidated and rebuilt. The classic mistake: putting COPY . . before RUN npm install. That means every code change triggers a full npm install. Fix: copy package.json and package-lock.json first, run install, then copy the rest. But even then, npm install can take 30 seconds. Use --cache-from to pull the last built image and reuse its layers. GitLab CI/CD makes this easy: docker build --cache-from $CI_REGISTRY_IMAGE:latest. This cuts build times from 5 minutes to 30 seconds for a typical Node app.

// io.thecodeforge — DevOps tutorial

FROM node:18-alpine AS builder
WORKDIR /app

# Copy only dependency files first to leverage cache
COPY package.json package-lock.json ./
RUN npm ci --only=production

# Now copy the rest of the app
COPY . .
RUN npm run build

FROM node:18-alpine
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["node", "dist/index.js"]

The Docker-in-Docker Trap: When Isolation Bites Back

Docker-in-Docker (dind) is the default way to build Docker images inside a GitLab CI job. You spin up a docker:dind service container, and your job talks to it via DOCKER_HOST=tcp://docker:2375. Sounds clean. But here's the catch: the dind container uses the overlay2 storage driver, which can eat disk space like candy. I've seen a single pipeline consume 50GB of disk because every build left dangling layers. The fix: run docker system prune -f in an after_script to clean up. Also, set DOCKER_DRIVER=overlay2 explicitly. Another trap: TLS. By default, dind expects TLS. If you don't set DOCKER_TLS_CERTDIR="", your job will fail with 'Cannot connect to the Docker daemon'. Always set that variable.

// io.thecodeforge — DevOps tutorial

# Safer dind configuration with cleanup
image: docker:24-git

services:
  - name: docker:24-dind
    command: ["--storage-driver", "overlay2"]

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
  DOCKER_DRIVER: overlay2

before_script:
  - docker info  # Verify daemon is reachable
  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  after_script:
    - docker system prune -f --all --volumes  # Free disk space
  only:
    - main

Secrets Management: Don't Bake Credentials Into Images

The worst thing you can do is hardcode a password in your Dockerfile. Next worst: passing it as a build arg that ends up in the image history. Use Docker build secrets (--secret) or GitLab CI/CD variables. For production, use a secrets manager like Vault or AWS Secrets Manager, and inject at runtime. Never, ever store secrets in the image. I've seen a startup's entire AWS account compromised because someone forgot to remove a .env file from the build context.

// io.thecodeforge — DevOps tutorial

# Use build secrets to avoid leaking credentials
FROM node:18-alpine
WORKDIR /app

# Copy only what's needed
COPY package.json package-lock.json ./
RUN npm ci --only=production

COPY . .

# Use a runtime secret, not a build arg
# The secret is mounted at /run/secrets/npm_token
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    npm publish --registry https://registry.npmjs.org/ --//registry.npmjs.org/:_authToken=$NPM_TOKEN

EXPOSE 3000
CMD ["node", "dist/index.js"]

Testing Your Docker Image in the Pipeline

Building an image is useless if you don't test it. The pattern: build the image, then run a container from it in the same pipeline, execute tests, and only push if tests pass. Use the docker run command with --rm to auto-clean. For integration tests, you might need to start dependent services (database, cache) as additional containers. GitLab CI's services keyword is perfect for this — it spins up containers linked to your job.

// io.thecodeforge — DevOps tutorial

# Build, test, then push
image: docker:24-git

services:
  - docker:dind
  - postgres:13-alpine  # For integration tests

variables:
  DOCKER_HOST: tcp://docker:2375
  DOCKER_TLS_CERTDIR: ""
  POSTGRES_PASSWORD: testpass
  POSTGRES_DB: testdb

stages:
  - build
  - test
  - push

build:
  stage: build
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:test
  artifacts:
    paths:
      - image_id.txt
  only:
    - main

test:
  stage: test
  script:
    - docker run --rm --network host -e DATABASE_URL=postgres://postgres:testpass@localhost:5432/testdb $CI_REGISTRY_IMAGE:test npm test
  only:
    - main

push:
  stage: push
  script:
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

When Docker in CI Is Overkill

If your project is a simple library or a static site, you don't need Docker in your pipeline. Use a pre-built image with the tools you need (e.g., node:18-alpine for Node, python:3.10 for Python). Run your tests directly. Only add Docker when you need to produce a container image. Also, if your team is small and your deployment target is a PaaS like Heroku or Vercel, they handle containerization for you. Don't add complexity you don't need.

// io.thecodeforge — DevOps tutorial

# Simple pipeline without Docker for a Node.js library
image: node:18-alpine

stages:
  - test
  - publish

test:
  stage: test
  script:
    - npm ci
    - npm test
  only:
    - main

publish:
  stage: publish
  script:
    - npm ci
    - npm publish
  only:
    - tags
  except:
    - branches

Kaniko Deep Dive

Docker-in-Docker requires privileged mode, which is a security concern for many organizations. Kaniko is Google's answer: it builds container images from a Dockerfile without requiring access to a Docker daemon at all. It runs as an unprivileged container, making it suitable for security-constrained environments like OpenShift or GitLab shared runners where privileged mode is disabled.

Kaniko works by extracting the base image filesystem, executing each Dockerfile command in userspace, and snapshotting the filesystem after each step. It doesn't use the Docker daemon, so there's no --cache-from in the Docker sense. Instead, Kaniko has its own cache mechanism: --cache=true caches intermediate layers in the registry, and --cache-ttl=24h controls expiration.

Important context: Google archived the original Kaniko repository in 2024. The community has forked it to continue maintenance. This means new features and security patches now come from the community fork, not Google. Before adopting Kaniko today, verify which fork is actively maintained and whether it supports your BuildKit-backed features (e.g., multi-stage builds work, but cache mounts don't).

When to choose Kaniko over dind or Buildah: Kaniko wins when you absolutely cannot run privileged containers. It loses on features (no BuildKit, no cache mounts) and speed (no parallel layer execution). For most teams, dind with proper cleanup is a better balance of security and performance.

// io.thecodeforge — DevOps tutorial

# Kaniko-based Docker build (no privileged mode needed)
image:
  name: gcr.io/kaniko-project/executor:debug
  entrypoint: [""]

variables:
  DOCKER_CONFIG: /kaniko/.docker
  KANIKO_CACHE: "true"
  KANIKO_CACHE_TTL: "24h"

before_script:
  - mkdir -p /kaniko/.docker
  - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n $CI_REGISTRY_USER:$CI_REGISTRY_PASSWORD | base64)\"}}}" > /kaniko/.docker/config.json

build:
  stage: build
  script:
    - /kaniko/executor
      --context $CI_PROJECT_DIR
      --dockerfile $CI_PROJECT_DIR/Dockerfile
      --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      --cache=true
      --cache-ttl=24h
      --skip-tls-verify=false
  only:
    - main

Buildah as Rootless Alternative

Buildah is a rootless container build tool from the Podman project. Unlike Kaniko (which runs as a single binary), Buildah provides a full command-line interface for building OCI-compliant images without a Docker daemon. It can run rootless, making it suitable for GitLab CI environments where privileged mode is restricted.

Buildah vs dind vs Kaniko: dind gives you the full Docker experience but requires privileged mode. Kaniko is truly daemonless but has a slower, less flexible build process. Buildah sits in the middle — it can run rootless, supports Dockerfiles (via buildah bud), and can directly push to OCI registries. It also supports building from Containerfiles (Podman's Dockerfile equivalent) and can mount host directories as build contexts.

However, Buildah has a steeper learning curve than dind or Kaniko. Its command syntax is different from Docker's, and some Dockerfile features are not fully supported. For teams already invested in the Podman ecosystem, Buildah is the natural choice. For Docker-native teams, the dind + proper cleanup pattern is usually simpler and better supported.

// io.thecodeforge — DevOps tutorial

# Rootless Buildah build (no privileged mode)
image: quay.io/buildah/stable

variables:
  STORAGE_DRIVER: vfs
  BUILDAH_ISOLATION: chroot

before_script:
  - buildah login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

build:
  stage: build
  script:
    - buildah bud
      -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
      -f Dockerfile .
    - buildah push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  only:
    - main

GitLab CI Security Scanning Templates

GitLab provides built-in security scanning templates that you can include in your pipeline with a single include directive. These templates add SAST (Static Application Security Testing), DAST (Dynamic Application Security Testing), container scanning, dependency scanning, and secret detection jobs to your pipeline without writing any custom scanning logic.

For container images, the Container-Scanning.gitlab-ci.yml template uses Trivy or Grype internally to scan your built image for vulnerabilities. It's enabled by adding include: - template: Jobs/Container-Scanning.gitlab-ci.yml to your pipeline. The scan runs after the build stage and produces a report that GitLab displays in the merge request UI.

For deeper control, you can integrate Trivy directly as a custom job. This allows you to configure severity thresholds, fail the pipeline on critical findings, and generate SBOMs. The Trivy template integration approach gives you the same scanning as the built-in GitLab template but with more flexibility in output format and exit behavior.

Key tip: always combine container scanning with dependency scanning. A vulnerability in a base image package is just as dangerous as one in your application dependencies.

// io.thecodeforge — DevOps tutorial

# GitLab CI with built-in container scanning
include:
  - template: Jobs/Container-Scanning.gitlab-ci.yml
  - template: Jobs/DAST.gitlab-ci.yml
  - template: Jobs/SAST.gitlab-ci.yml
  - template: Jobs/Secret-Detection.gitlab-ci.yml
  - template: Jobs/Dependency-Scanning.gitlab-ci.yml

stages:
  - build
  - test
  - scan
  - deploy

variables:
  SECURE_LOG_LEVEL: info
  DS_DISABLE_TELEMETRY: "true"
  CS_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build:
  stage: build
  image: docker:24-git
  services:
    - docker:dind
  variables:
    DOCKER_HOST: tcp://docker:2375
    DOCKER_TLS_CERTDIR: ""
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker build -t $CS_IMAGE .
    - docker push $CS_IMAGE
  only:
    - main

# Container scanning runs automatically with the CS_IMAGE variable set
# Fail pipeline on critical vulnerabilities
container_scanning:
  variables:
    CS_SEVERITY_THRESHOLD: critical
    CS_FEATURE_DISABLE: ""
  allow_failure: false

DAG Pipelines with Needs and Workflow Rules

Not all jobs in a pipeline need to run sequentially. GitLab's Directed Acyclic Graph (DAG) pipelines let you run independent jobs in parallel using the needs keyword, dramatically reducing total pipeline time. Without DAG, GitLab runs jobs stage-by-stage: all jobs in build finish before any job in test starts. With needs, you specify exact dependencies — a test job can start as soon as its build job finishes, even if other build jobs are still running.

The workflow:rules keyword prevents duplicate pipelines. The classic problem: a push to a branch triggers both a branch pipeline and a merge request pipeline. workflow:rules with if: $CI_PIPELINE_SOURCE == 'merge_request_event' ensures only the MR pipeline runs, saving CI minutes.

Multi-project pipelines let you trigger pipelines in other projects. For example, a microservice build can trigger the deployment project's pipeline. This is essential for organizations with separate repos for application code and infrastructure. Use trigger job keyword with a project path and optional branch.

// io.thecodeforge — DevOps tutorial

# DAG pipeline with needs and workflow rules

workflow:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == "main"

stages:
  - build
  - test
  - scan
  - deploy

variables:
  IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

build-amd64:
  stage: build
  script:
    - docker build --platform linux/amd64 -t $IMAGE_TAG-amd64 .
    - docker push $IMAGE_TAG-amd64

build-arm64:
  stage: build
  script:
    - docker build --platform linux/arm64 -t $IMAGE_TAG-arm64 .
    - docker push $IMAGE_TAG-arm64

# Runs as soon as BOTH amd64 builds finish
# Does NOT wait for arm64 build
test-amd64:
  stage: test
  needs: ["build-amd64"]
  script:
    - docker run --rm $IMAGE_TAG-amd64 npm test

# Runs in parallel with test-amd64
test-arm64:
  stage: test
  needs: ["build-arm64"]
  script:
    - docker run --rm $IMAGE_TAG-arm64 npm test

# Runs after both test jobs finish (not waiting for scan stage)
deploy:
  stage: deploy
  needs: ["test-amd64", "test-arm64"]
  script:
    - docker buildx imagetools create -t $CI_REGISTRY_IMAGE:latest $IMAGE_TAG-amd64 $IMAGE_TAG-arm64

# Multi-project pipeline trigger (separate repo)
deploy-prod:
  stage: deploy
  needs: ["test-amd64", "test-arm64"]
  trigger:
    project: myorg/deployment-manifests
    branch: main
    strategy: depend

GitLab Container Registry Lifecycle

The GitLab Container Registry is a Docker-compatible registry built into every GitLab project. It's convenient — no separate registry to set up — but left unmanaged, it fills up with stale images. Every pipeline push creates a new image tag. Over months, you accumulate thousands of images consuming storage and making the registry slow to browse.

GitLab's cleanup policy is the solution. It automatically deletes tags based on rules you define: keep the latest N tags (e.g., 10), delete tags older than N days (e.g., 30), delete tags matching a regex pattern (e.g., .*-dev$), and crucially, delete untagged artifacts. Untagged artifacts are dangling layers from interrupted pushes or failed cleanup jobs — they take up space but are invisible in the UI.

The deep-dive setting most teams miss: DELETE_MULTI_TAGGED_IMAGES=false (default). This prevents deletion of images that have multiple tags pointing to the same digest. For example, if latest and v1.2.3 point to the same image, setting this to true might delete the image when v1.2.3 is cleaned up, breaking the latest tag. Keep this false unless you understand the implications.

For GitLab SaaS, cleanup policies run once per day. For self-managed, you can configure the frequency. Regardless, periodically verify your policy is working by checking the registry's storage usage in the project settings.

// io.thecodeforge — DevOps tutorial

# Container registry cleanup policy configuration
# Set in Project > Settings > CI/CD > Container Registry

# Example policy (configured via GitLab UI or API):
cleanup_policy:
  cadence: "0 0 * * *"  # Daily at midnight
  keep_n: 10            # Keep last 10 tags per image name
  older_than: 30d       # Delete tags older than 30 days
  name_regex_delete:
    - ".*-dev$"        # Delete all dev tags
    - ".*-test$"       # Delete all test tags
  name_regex_keep:
    - ".*-stable$"     # Always keep stable tags
    - "^v[0-9]+\.[0-9]+\.[0-9]+$"  # Keep semver tags
  delete_multi_tagged_images: false  # Don't delete images with multiple tags
  delete_untagged_artifacts: true    # Delete orphaned layers