Docker Image Security Scanning: Stop Shipping Vulnerable Containers to Production

You just pushed a Docker image to production. Two hours later, your security team calls: 'We found a critical CVE in your base image.' Now you're scrambling to rebuild, retest, and redeploy. Sound familiar? Docker image security scanning isn't optional anymore — it's the difference between a routine deploy and a breach notification.

The problem is simple: your image is a stack of layers, each potentially carrying vulnerable packages. Base images like node:18 or python:3.11 are updated frequently, but your Dockerfile might pin an old version. Without scanning, you're blind to known vulnerabilities until it's too late.

By the end of this, you'll be able to integrate vulnerability scanning into your Docker build pipeline, interpret scan results like a pro, and avoid the gotchas that burn teams in production. You'll know which tool to use when, how to handle false positives, and how to set up automated gates that stop vulnerable images from ever reaching your registry.

Why You Can't Trust Your Base Image

Every Docker image starts with a base — FROM node:18, FROM python:3.11-slim, FROM alpine:3.18. These images are maintained by communities, but they're not immune to vulnerabilities. A single outdated package in the base can expose your entire application.

Consider this: the node:18 image contains over 100 OS packages (glibc, openssl, zlib, etc.). Each has a version string that maps to CVEs. If you don't scan, you're flying blind. The worst part? Base images are updated frequently, but your Dockerfile might pin a specific tag like node:18 which points to a moving target. Today's node:18 is not tomorrow's node:18.

The fix: always pin base images by digest (sha256:...) and scan them before building your app layer. This gives you reproducible builds and a known vulnerability baseline.

// io.thecodeforge — DevOps tutorial

# Bad: tag-based, changes without notice
# FROM node:18

# Good: digest-based, reproducible
FROM node:18@sha256:abc123def456...

# Now scan this base image before adding your code
# $ trivy image node:18@sha256:abc123def456...

WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
EXPOSE 3000
CMD ["node", "server.js"]

Choosing the Right Scanner: Trivy vs Grype vs Docker Scout

You have options. Trivy (Aqua Security) is the Swiss Army knife — scans OS packages, language-specific deps, IaC misconfigurations, and even Kubernetes manifests. Grype (Anchore) focuses on container images and integrates tightly with Syft for SBOM generation. Docker Scout is Docker's own offering, baked into Docker Desktop and Hub.

Which one should you use? If you're already in the Docker ecosystem and want zero-config, Docker Scout is tempting. But it's a SaaS product — your image data leaves your network. For air-gapped environments or strict compliance, Trivy or Grype are better. Trivy wins on speed and breadth: it can scan a 1GB image in under 30 seconds. Grype is slower but produces more detailed SBOMs.

My take: use Trivy for CI/CD gates (fast, reliable) and Grype for compliance audits (detailed SBOM). Docker Scout is fine for local dev awareness but don't rely on it as your only gate.

// io.thecodeforge — DevOps tutorial

# Install Trivy (macOS)
brew install trivy

# Install Grype
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh | sh -s -- -b /usr/local/bin

# Scan with Trivy (fast)
trivy image --severity CRITICAL,HIGH --no-progress myapp:latest

# Scan with Grype (detailed SBOM)
grype myapp:latest --only-fixed --fail-on high

# Docker Scout (requires Docker Desktop)
docker scout quickview myapp:latest

Integrating Scanning into CI/CD: The Gate That Actually Works

A scan that doesn't block the pipeline is just a report nobody reads. You need a hard gate: if the image has any CRITICAL vulnerability with a known fix, fail the build. But beware — not all CVEs are equal. Some are in unused packages, some have no exploit in the wild, and some are in test dependencies.

The trick: use --fail-on with a severity threshold, but also maintain a .trivyignore for accepted risks. Every ignored CVE must have a comment with a ticket number and expiration date. This prevents 'ignore and forget'.

Here's a production-grade CI step for GitHub Actions that scans, fails on critical/high, and allows exemptions.

// io.thecodeforge — DevOps tutorial

name: Docker Security Scan

on:
  push:
    branches: [main]
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build Docker image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: 'myapp:${{ github.sha }}'
          format: 'sarif'
          output: 'trivy-results.sarif'
          severity: 'CRITICAL,HIGH'
          exit-code: '1'  # Fail on any critical/high
          ignore-unfixed: true  # Only fail on vulnerabilities with a fix

      - name: Upload Trivy results to GitHub Security tab
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: 'trivy-results.sarif'

      - name: Fail if critical vulnerabilities older than 30 days
        run: |
          # Custom check: fail if any critical CVE has been known for >30 days
          trivy image --severity CRITICAL --format json myapp:${{ github.sha }} | \
            jq -e '.Results[].Vulnerabilities[] | select(.PublishedDate < (now - 30*86400))' > /dev/null && \
            echo "Found old critical CVE" && exit 1 || echo "All critical CVEs are recent"

Handling False Positives and Ignored CVEs

No scanner is perfect. You'll get false positives — a CVE in a library you don't use, or a vulnerability that's only exploitable with a specific configuration you don't have. Ignoring them is fine, but do it right.

Create a .trivyignore file in your repo. Each line is a CVE ID followed by a comment. The comment must include a ticket number and an expiration date. This forces periodic review. If the CVE is still unfixed after 90 days, your team must re-evaluate.

Example: CVE-2024-1234 # OPS-5678: Accepted risk until 2024-06-01. Library not loaded at runtime.

Never ignore a CVE without a ticket. That's how vulnerabilities become 'technical debt' that never gets paid.

// io.thecodeforge — DevOps tutorial

# Accepted risks - review quarterly
CVE-2024-1234 # OPS-5678: Accepted until 2024-06-01. Library not loaded at runtime.
CVE-2023-4567 # OPS-5679: Accepted until 2024-09-01. No known exploit in the wild.

# False positive - not applicable to our architecture
CVE-2022-7890 # OPS-5680: False positive. Only affects Windows builds.

Scanning During Build: Multi-Stage and Distroless Images

Multi-stage builds are great for reducing image size, but they complicate scanning. You need to scan the final stage, not the builder stage. The builder stage might have compilers and dev tools with CVEs, but they're not in the final image.

Distroless images (e.g., gcr.io/distroless/nodejs) are minimal — no shell, no package manager. This reduces the attack surface dramatically. But they also make scanning harder because there's no package database to query. Trivy handles this by scanning the binary layers directly.

Best practice: scan both the builder stage (for awareness) and the final stage (for the gate). Fail only on the final stage.

// io.thecodeforge — DevOps tutorial

# Build stage
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Production stage
FROM gcr.io/distroless/nodejs18-debian11
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
EXPOSE 3000
CMD ["dist/server.js"]

# Scan the final stage:
# $ trivy image myapp:latest
# Note: distroless images have no package manager, but Trivy scans binaries.

Secrets Scanning: The CVE Nobody Talks About

Vulnerabilities are one thing, but hardcoded secrets in your image are a direct line to your infrastructure. Docker images are often shared via registries — if you accidentally include an AWS key or a database password, anyone with pull access can use it.

Trivy has a built-in secrets scanner (--scanners secret). It checks for patterns like AWS keys, GitHub tokens, private keys, and more. Run it as part of your scan. The output is clear: file path, line number, and the type of secret.

Never build an image with secrets. Use Docker build secrets (--secret) or multi-stage builds to avoid copying sensitive files into the final image.

// io.thecodeforge — DevOps tutorial

# Scan for secrets in the image
trivy image --scanners secret myapp:latest

# Example output:
# 2024-01-15T10:00:00Z    INFO    Secret scanning is enabled
# Total: 1 secret
# ┌─────────────────────┬──────────────────┬──────────┐
# │ File                │ Secret Type      │ Severity │
# ├─────────────────────┼──────────────────┼──────────┤
# │ /app/.env           │ AWS Access Key   │ CRITICAL │
# └─────────────────────┴──────────────────┴──────────┘

# Fix: remove the .env file from the image. Use Docker secrets or env vars at runtime.

Scanning in Air-Gapped Environments

If your production environment has no internet access (air-gapped), you can't download vulnerability databases on the fly. You need to pre-download the DB and make it available to the scanner.

Trivy supports offline mode. Download the DB on a connected machine, then transfer it to the air-gapped environment. Use --cache-dir to point to the local DB. Update the DB periodically (e.g., weekly) and re-scan your images.

Grype has a similar approach with its grype db update command. Both tools allow you to specify a local DB path.

// io.thecodeforge — DevOps tutorial

# On connected machine: download Trivy DB
trivy image --download-db-only --cache-db /tmp/trivy-db

# Copy /tmp/trivy-db to air-gapped environment (e.g., via USB)

# On air-gapped machine: scan using local DB
trivy image --cache-dir /path/to/trivy-db --skip-db-update myapp:latest

# For Grype:
# Download DB: grype db update -o /tmp/grype-db
# Scan: grype myapp:latest --db /tmp/grype-db

When Not to Use Image Scanning (And What to Do Instead)

Image scanning is not a silver bullet. It won't catch zero-day vulnerabilities (no known CVE yet), logic flaws in your application, or misconfigurations in your runtime environment. It also won't help if your base image is completely custom and not in any CVE database.

For zero-days, you need runtime security monitoring (e.g., Falco, Sysdig). For application logic, you need SAST/DAST tools. For misconfigurations, use Kubernetes security policies (OPA, Kyverno).

Also, scanning every image in a large registry daily can be expensive. Prioritize: scan production images weekly, development images on every build. Use differential scanning — only scan layers that changed since the last scan.

Cosign Attestation Integration — Attach Scan Results as Signed Attestations

Cosign, part of the Sigstore project, supports attaching arbitrary attestations to container images as OCI artifacts. You can attach vulnerability scan reports, SBOMs, test results, or SLSA provenance statements as signed attestations. The attestation is stored alongside the image in the registry and can be verified at admission time.

To integrate scan results with Cosign: run your vulnerability scanner (e.g., trivy image --format cosign-vuln), pipe the output to cosign attest, and attach to the image. The attestation includes the scan results in a standard in-toto predicate format. At admission time, Kyverno or a policy engine verifies both the image signature and the attestation — ensuring the image was scanned and passed policy before deployment.

SLSA (Supply-chain Levels for Software Artifacts) provenance attestations track where and how an image was built. A SLSA provenance attestation includes: build platform, build command, source repository, and builder identity. When combined with vulnerability scan attestations, you get a complete security pedigree for every image: who built it, how it was built, and what vulnerabilities it had at build time.

Cosign attestations support keyless signing via Fulcio and OIDC (OpenID Connect). In CI/CD, the build system (GitHub Actions, GitLab CI) obtains a short-lived certificate from Fulcio, signs the attestation, and attaches it to the image. No long-lived keys to manage or rotate. The attestation's validity can be verified against the OIDC token — proving it was created by your CI pipeline.

#!/bin/bash
# Attach scan results as Cosign attestation

# ── 1. Scan image with Trivy, output in Cosign format ──────────
trivy image --format cosign-vuln --output vuln.json myapp:latest

# ── 2. Attach the vulnerability report as an attestation ───────
cosign attest --predicate vuln.json \
  --key cosign.key \
  myapp:latest

# ── 3. Verify the attestation ──────────────────────────────────
cosign verify-attestation --key cosign.pub myapp:latest

# ── 4. Verify with policy (filter for vulnerability assertions) ─
cosign verify-attestation --key cosign.pub \
  --type vuln myapp:latest

# ── Keyless (no key management) ────────────────────────────────
# In CI/CD (GitHub Actions):
cosign attest --predicate vuln.json \
  myapp:latest
# Uses OIDC token from GitHub — no key file needed

# ── SLSA provenance ───────────────────────────────────────────
cosign attest --predicate slsa-provenance.json \
  --type slsa.dev/provenance/v1 \
  myapp:latest

Kyverno Admission Controller for Image Security — Block Vulnerable Images at Deploy Time

Kyverno's ImageValidatingPolicy can enforce image security policies at Kubernetes admission time. Beyond signature verification, you can block pods that use images with known critical vulnerabilities, enforce that images come only from approved registries, or require that all images have a valid vulnerability scan attestation.

Combine with Cosign attestations for maximum coverage: policy requires (1) the image is from an approved registry, (2) the image has a valid Cosign signature, (3) the image has a vulnerability scan attestation with no critical CVEs, (4) the image has a SLSA provenance attestation showing it was built in your CI pipeline.

Deploy in Audit mode first to discover which workloads would be blocked. Kyverno reports policy violations in the resource's status and in Kyverno logs. After reviewing, switch to Enforce mode.

# io.thecodeforge — devops tutorial

apiVersion: kyverno.io/v2beta1
kind: ImageValidatingPolicy
metadata:
  name: require-vuln-attestation
spec:
  rules:
    - name: verify-no-critical-cves
      imageReferences:
        - "*"
      verification:
        attestors:
          - entries:
              - keys:
                  publicKeys: |-
                    -----BEGIN PUBLIC KEY-----
                    MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE...
                    -----END PUBLIC KEY-----
        attestations:
          - predicateType: https://cosign.sigstore.dev/attestation/vuln/v1
            conditions:
              - expression: "predicate.scanner.result < 1"
      failureAction: Enforce

---
apiVersion: kyverno.io/v2beta1
kind: ClusterPolicy
metadata:
  name: require-trusted-registry
spec:
  rules:
    - name: check-registry
      match:
        any:
          - resources:
              kinds:
                - Pod
      celPreconditions:
        - expression: "!image.startsWith('registry.example.com/')"
      validate:
        cel:
          expressions:
            - expression: "false"
              message: "Images must come from registry.example.com"

Notary v2 (Notation) — OCI-Compliant Image Signing Alternative to Cosign

Notation is the reference implementation of Notary v2, an OCI-compliant image signing standard. Unlike Cosign (which stores signatures as OCI artifacts in a separate repository), Notation embeds signatures directly into the OCI manifest using the OCI 1.1 referrers API. This means the signature travels with the image — no need to manage separate signature repositories or worry about signature GC.

Setting up Notation: generate a self-signed certificate or use an existing enterprise CA. Add it to a trust store (notation cert add). Sign images with notation sign --key . Verify with notation verify .

Notation integration with Kyverno: Kyverno's ImageValidatingPolicy supports Notation attestors natively. You specify trust stores and trust policies instead of Cosign public keys. This is useful for organizations that already operate an X.509 PKI.

Both Notation and Cosign achieve the same goal: proving image integrity and origin. The choice depends on your key management infrastructure. If you have an existing PKI, Notation is the natural fit. If you prefer OIDC-based keyless signing, use Cosign.

#!/bin/bash
# Notary v2 (Notation) setup and signing

# ── 1. Install Notation ────────────────────────────────────────
# macOS: brew install notation
# Linux: curl -LO https://github.com/notaryproject/notation/releases/...

# ── 2. Create a certificate (self-signed, or use enterprise CA) ─
notation cert generate-test --default "registry.example.com"

# ── 3. Sign the image ──────────────────────────────────────────
notation sign --key "registry.example.com" registry.example.com/myapp:latest

# ── 4. Verify the signature ────────────────────────────────────
notation verify registry.example.com/myapp:latest
# Output: Successfully verified

# ── 5. List signatures ────────────────────────────────────────
notation ls registry.example.com/myapp:latest

# ── 6. Kyverno policy integration ──────────────────────────────
# In ImageValidatingPolicy, use:
# verification:
#   attestors:
#     - entries:
#         - notations:
#             certs: "notation-trust-store"
#             trustPolicy: "notation-trust-policy"

Registry-Level Scanning with Harbor — Block Vulnerable Pushes at the Registry

Harbor is an open-source cloud-native registry that integrates vulnerability scanning directly into the registry workflow. When an image is pushed to Harbor, it automatically triggers a scan (using Trivy, Grype, or Aqua CSP). Harbor evaluates the scan results against policy: images with critical vulnerabilities can be blocked from being pulled, or the push itself can be rejected.

Harbor integrates with Trivy natively (built-in scanner, no configuration needed). It also supports custom scanners via the Scanner Adapter interface (Snyk, Aqua, Anchore). The scan results are displayed in the Harbor UI and exposed via API for integration with CI/CD pipelines.

Setup: deploy Harbor, enable the Trivy scanner, and create a vulnerability policy. The policy specifies the severity threshold (e.g., block if any CRITICAL), whether to scan on push, and the action (block push, block pull, or quarantine). Harbor also supports image replication with scan results — only replicate images that pass policy to your production registry.

Comparison with CI/CD scanning: CI/CD scanning blocks vulnerable images before they reach the registry. Registry-level scanning catches images that bypass CI (e.g., manual pushes, images from external pipelines, or base image drift after rebuilding). Both are necessary for defense in depth.

# io.thecodeforge — devops tutorial

# Harbor vulnerability policy (configured in UI or via API)
# This blocks pull of images with critical or high vulnerabilities

apiVersion: goharbor.io/v1alpha1
kind: HarborProject
metadata:
  name: production
spec:
  vulnerabilityScanPolicy:
    analyzer: trivy
    parameters:
      severity: critical,high
      fixable: true
  enforcement:
    pull:
      enabled: true
      severity: critical,high
    push:
      enabled: false  # Allow push, but block pull
  scanOnPush: true

# API equivalent:
# curl -X PUT "https://harbor.example.com/api/v2.0/projects/production" \
#   -H "Content-Type: application/json" \
#   -d '{"vulnerability_scan_policy": {"analyzer": "trivy", "parameters": {"severity": "critical,high"}}}'

SBOM Format Comparison — CycloneDX vs SPDX and Legal Requirements

Choosing the right SBOM format depends on your use case: security operations, vulnerability management, or legal/license compliance. Both CycloneDX and SPDX are ISO/IEC standards, but they differ in scope and data model.

CycloneDX (OWASP) — ISO/IEC 19770-2:2024. Designed for application security and vulnerability management. Key features: - Component metadata: supplier, version, hashes, licenses - Dependency graph: explicit relationships between components - Vulnerability references: links to CVE, GHSA, and other advisory databases - Services and external references - Native support in Trivy, Syft, and Docker Scout - Best for: CI/CD security gates, vulnerability correlation, operations

SPDX (Linux Foundation) — ISO/IEC 5962:2021. Originally designed for license compliance. Key features: - File-level granularity (can list every file in a package) - Detailed license information (per-file and per-package) - Cross-reference document relationships - More verbose (3-5x larger than CycloneDX) - Best for: legal audits, M&A diligence, license compliance

NTIA Minimum Elements (EO 14028): Both formats can satisfy the six NTIA minimum elements: (1) Supplier name, (2) Component name, (3) Version, (4) Dependency relationships, (5) Author, (6) Timestamp. CycloneDX maps these naturally via supplier, name, version, dependencies, author, and timestamp fields. SPDX requires more fields but covers the same elements.

Recommendation: Generate CycloneDX for security workflows (vulnerability scanning, incident response) and SPDX for legal/compliance workflows. Most SBOM tools (Syft, Trivy) can generate both from a single scan. Store both in the registry as OCI attestations. For EO 14028 compliance, CycloneDX is the more practical choice — smaller, security-focused, and directly integrated with vulnerability databases.

#!/bin/bash
# Generate both CycloneDX and SPDX from the same image

# ── CycloneDX (JSON) — for security workflows ──────────────────
syft myapp:latest -o cyclonedx-json > sbom.cdx.json

echo "CycloneDX size: $(wc -c < sbom.cdx.json) bytes"
echo "Components: $(jq '.components | length' sbom.cdx.json)"

# ── SPDX (JSON) — for legal compliance ────────────────────────
syft myapp:latest -o spdx-json > sbom.spdx.json

echo "SPDX size: $(wc -c < sbom.spdx.json) bytes"
echo "Files: $(jq '.files | length' sbom.spdx.json)"

# ── Compare NTIA minimum elements ─────────────────────────────
# CycloneDX:
jq '.metadata.supplier.name' sbom.cdx.json  # Supplier name
jq '.metadata.timestamp' sbom.cdx.json        # Timestamp

# SPDX:
jq '.creationInfo.creators' sbom.spdx.json   # Author
jq '.creationInfo.created' sbom.spdx.json     # Timestamp