Docker Secrets Management: Stop Hardcoding Credentials in Production

I've seen a startup's entire database credentials leaked because someone ran docker inspect on a running container. The password was in an environment variable. That's not a rookie mistake — it's a design flaw. Docker Secrets Management exists precisely to prevent this. It's not about convenience; it's about not getting pwned at 3 AM when an attacker enumerates your containers. After this article, you'll be able to design a secret delivery pipeline that survives audits, rotates keys without downtime, and doesn't make you cringe when ops asks for the secret inventory.

Why Environment Variables Are a Security Nightmare

Environment variables are the default way to pass configuration to containers. They're easy, they're documented everywhere, and they're wrong for secrets. Here's why: any process with access to /proc//environ can read them. That includes docker inspect, kubectl exec, and any compromised container on the same host. Worse, env vars leak into logs, error reports, and core dumps. I've personally debugged a case where a Python traceback printed the entire DB password because it was in the environment. The fix? Never put secrets in env vars in production. Use Docker secrets or a vault.

// io.thecodeforge — DevOps tutorial

# Run a container with a secret as env var
docker run -d --name leaky -e DB_PASSWORD=supersecret alpine sleep 3600

# Inspect the container — password visible
docker inspect leaky | grep DB_PASSWORD
# Output: "DB_PASSWORD=supersecret"

# Or read from /proc
docker exec leaky cat /proc/1/environ | tr '\0' '\n' | grep DB_PASSWORD
# Output: DB_PASSWORD=supersecret

Docker Swarm Secrets: The Built-In Solution

Docker Swarm has native secret management. Secrets are encrypted at rest in the Swarm store and only decrypted when a container that needs them is scheduled. They're mounted as files in /run/secrets/ by default, with in-memory tmpfs. No disk writes. No env var leaks. The flow: docker secret create stores the secret, then you grant access to services via --secret. The container reads the file at startup. Rotation requires updating the secret and redeploying the service. For zero-downtime rotation, you can mount multiple versions and have the app watch for changes.

// io.thecodeforge — DevOps tutorial

# Create a secret from stdin
echo "my-db-password" | docker secret create db_password -

# Verify
docker secret ls

# Deploy a service with the secret
docker service create --name web \
  --secret db_password \
  --publish 80:80 \
  nginx:alpine

# Inside the container, the secret is at /run/secrets/db_password
docker exec $(docker ps -q) cat /run/secrets/db_password
# Output: my-db-password

Kubernetes Secrets: More Power, More Pitfalls

Kubernetes Secrets are similar but more flexible — and more dangerous. By default, Secrets are stored unencrypted in etcd. You must enable encryption at rest. They can be mounted as volumes or injected as env vars (don't do the latter). The real power is in integration with external vaults like HashiCorp Vault via CSI drivers or sidecars. For production, never use plain Kubernetes Secrets for anything sensitive. Always enable encryption and use RBAC to restrict access. The classic mistake: storing a cloud API key in a Secret and then accidentally committing the YAML to git.

// io.thecodeforge — DevOps tutorial

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: bXktZGItcGFzc3dvcmQ=  # base64 of 'my-db-password'
---
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
  - name: web
    image: nginx:alpine
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: db-credentials

Secret Rotation Without Downtime

Rotating secrets is where most teams fail. The naive approach: update the secret, restart all containers. That causes downtime. Better: mount the secret as a file and have the application watch for file changes (e.g., inotify). When the file changes, reload the connection. For databases, use a connection pool that supports credential refresh (like HikariCP with a DataSource that reads from a file). In Swarm, you can update a secret and redeploy the service with --update-delay for rolling updates. In Kubernetes, use a controller that watches the Secret and triggers a rolling restart via annotation change.

// io.thecodeforge — DevOps tutorial

import os
import time
import signal

SECRET_PATH = '/run/secrets/db_password'

def load_secret():
    with open(SECRET_PATH) as f:
        return f.read().strip()

def reload_handler(signum, frame):
    global db_password
    db_password = load_secret()
    print(f"Secret reloaded at {time.ctime()}")

# Register SIGHUP handler for manual reload
signal.signal(signal.SIGHUP, reload_handler)

# Initial load
db_password = load_secret()
print(f"Initial password: {db_password}")

# Simulate main loop
while True:
    time.sleep(10)
    # In production, use inotify or watchdog library for file change detection

External Vault Integration: HashiCorp Vault with Docker

For serious production environments, built-in secrets managers are too limited. You need audit logs, dynamic secrets (short-lived credentials), and fine-grained access control. HashiCorp Vault is the gold standard. Integrate with Docker via the Vault agent sidecar pattern: a container that authenticates to Vault, retrieves secrets, and writes them to a shared volume. The application reads from that volume. Vault handles rotation automatically. The downside: complexity. You now manage Vault itself. But for compliance-heavy environments (PCI-DSS, HIPAA), it's non-negotiable.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  vault-agent:
    image: vault:latest
    command: agent -config=/etc/vault/config.hcl
    volumes:
      - shared-secrets:/secrets
      - ./vault-config:/etc/vault
    environment:
      VAULT_ADDR: https://vault.example.com
  app:
    image: myapp:latest
    volumes:
      - shared-secrets:/secrets:ro
    depends_on:
      - vault-agent
volumes:
  shared-secrets:

When Not to Use Docker Secrets

Docker secrets are overkill for local development. Use .env files or docker-compose's env_file — just don't commit them to git. For single-node deployments, bind-mounting a secrets directory with restricted permissions is simpler and equally secure. Also, if your orchestrator doesn't support secrets (e.g., plain Docker Compose without Swarm), you're stuck with workarounds. In those cases, use a vault sidecar or encrypt the env var with a key that the app decrypts at startup. The key itself must be stored securely — a chicken-and-egg problem that vault solves.

// io.thecodeforge — DevOps tutorial

# For local dev, just mount a file with restricted permissions
mkdir -p ./secrets
echo "dev-password" > ./secrets/db_password
chmod 0400 ./secrets/db_password
chown 1000:1000 ./secrets/db_password

docker run -v $(pwd)/secrets:/run/secrets:ro alpine cat /run/secrets/db_password
# Output: dev-password

swarm-external-secrets Plugin Deep Dive

The swarm-external-secrets Docker plugin brings multi-provider secret management to Docker Swarm. Instead of storing secrets in the Swarm Raft store, it fetches them on-demand from external providers: HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and OpenBao. Authentication is handled via AppRole for Vault/OpenBao, IAM roles for AWS, and managed identities for Azure. A key feature is SHA256 hash-based rotation detection — the plugin periodically compares the hash of the stored secret against the current value in the external store; when they differ, it rotates without service redeployment. The plugin also performs cleanup of old plugin versions to prevent version drift across the Swarm. However, a limitation to be aware of: worker nodes cannot serve as plugin managers; only manager nodes coordinate secret retrieval, which adds latency in large clusters and creates a potential bottleneck if the manager pool is undersized.

// io.thecodeforge — DevOps tutorial

# Install the plugin on all Swarm nodes
docker plugin install grafana/swarm-external-secrets:latest --alias secrets

# Configure Vault provider in /etc/docker/secrets-plugin.yaml
providers:
  vault:
    addr: https://vault.example.com:8200
    approle:
      role_id: "your-role-id"
      secret_id: "your-secret-id"
    tls_skip_verify: false

# Create a secret backed by external provider
docker secret create --driver secrets \
  --driver-opts provider=vault,path=secret/data/db/password \
  db_password -

# Deploy service — secret retrieved from Vault at container start
docker service create --name web \
  --secret db_password \
  nginx:alpine

Zero-Downtime Secret Rotation Patterns

Rotating secrets without dropping connections is the holy grail of secrets management. There are two main patterns: application-level reload and container restart. Application-level reload involves the app watching the secret file via inotify or polling, then updating in-memory credentials without restarting. This requires the app to support credential refresh — not all do (e.g., legacy Java apps). The restart approach uses Docker's update_config settings with rolling updates: --update-delay 10s --update-parallelism 1 --update-order start-first. This gradually replaces containers, ensuring capacity at all times. Rotation interval tuning is critical — too fast and you overwhelm the system; 5 minutes is the standard production interval for most setups. For monitoring rotation events, pipe Docker logs into Loki: docker service logs --since 5m | promtail. This gives you a searchable audit trail of when secrets were rotated and which containers restarted as a result.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  api:
    image: myapp:latest
    secrets:
      - db_password_v2
    deploy:
      replicas: 5
      update_config:
        parallelism: 1
        delay: 10s
        order: start-first
        failure_action: rollback
      restart_policy:
        condition: any

secrets:
  db_password_v2:
    external: true

BuildKit Build-Time Secrets for Non-Swarm Docker

Docker BuildKit supports build-time secrets that never end up in the final image. Using --mount=type=secret during a docker build, you can pass sensitive values (npm tokens, SSH keys, signing certs) to the build process without embedding them. The syntax is --mount=type=secret,id=mysecret in the Dockerfile RUN command, then --secret id=mysecret,src=./local-secret on the docker build command line. For docker-compose builds, use the secrets key under the build context with a secrets section in your compose file (v3.9+). The critical difference from Swarm secrets: BuildKit secrets exist only during the build phase and are never present in the running container. Swarm secrets are for runtime. They serve different purposes — build vs run. A common trap: developers put deployment credentials in build args (--build-arg) which DO end up in image history. BuildKit secrets do not. Always choose --mount=type=secret over build args for anything sensitive.

// io.thecodeforge — DevOps tutorial

# syntax=docker/dockerfile:1
FROM node:18-alpine

WORKDIR /app
COPY package*.json ./

# Mount npm token as build-time secret — not stored in image
RUN --mount=type=secret,id=npmrc \
    cat /run/secrets/npmrc > .npmrc && \
    npm ci --only=production

COPY . .
CMD ["node", "server.js"]

# Build command:
# docker build --secret id=npmrc,src=./.npmrc -t myapp .

Secrets in CI/CD

CI/CD pipelines have unique secret challenges: secrets must be available during build but must never leak into logs, artifacts, or image layers. GitHub Actions offers encrypted secrets per repo/environment accessible via ${{ secrets.MY_SECRET }}. GitLab CI has CI/CD Variables with masking and protection (masked variables are hidden in job logs). The perennial debate: mount secrets as files or inject as env vars? Files win — env vars are visible in /proc, printed by debugging tools, and logged by CI runners. Use GitHub Actions to write secrets to a temp file with restricted permissions. For GitLab, use the file type variable (variable type: file) which mounts as a file. To prevent secret leak in build logs: never echo secrets, disable verbose commands, use set +x before secret access, and scan logs for accidental exposure with tools like truffleHog or git-secrets. Most importantly, never cache layers that contain secret data — use --no-cache or target specific stages that exclude secrets.

// io.thecodeforge — DevOps tutorial

name: Build with Secrets
on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # Write secret to file — never to env var
      - name: Write npmrc
        run: |
          mkdir -p ~/.npm
          echo "//registry.npmjs.org/:_authToken=${{ secrets.NPM_TOKEN }}" > ~/.npmrc
          chmod 0600 ~/.npmrc
        shell: bash

      - name: Build with BuildKit
        run: |
          docker build --secret id=npmrc,src=$HOME/.npmrc -t myapp .

      - name: Scan for leaks
        run: |
          docker history --no-trunc myapp | grep -i token && exit 1 || echo "No tokens in history"