Docker Secrets Management: Stop Hardcoding Credentials in Production

I've seen a startup's entire database credentials leaked because someone ran docker inspect on a running container. The password was in an environment variable. That's not a rookie mistake — it's a design flaw. Docker Secrets Management exists precisely to prevent this. It's not about convenience; it's about not getting pwned at 3 AM when an attacker enumerates your containers. After this article, you'll be able to design a secret delivery pipeline that survives audits, rotates keys without downtime, and doesn't make you cringe when ops asks for the secret inventory.

Why Environment Variables Are a Security Nightmare

Environment variables are the default way to pass configuration to containers. They're easy, they're documented everywhere, and they're wrong for secrets. Here's why: any process with access to /proc/<pid>/environ can read them. That includes docker inspect, kubectl exec, and any compromised container on the same host. Worse, env vars leak into logs, error reports, and core dumps. I've personally debugged a case where a Python traceback printed the entire DB password because it was in the environment. The fix? Never put secrets in env vars in production. Use Docker secrets or a vault.

// io.thecodeforge — DevOps tutorial

# Run a container with a secret as env var
docker run -d --name leaky -e DB_PASSWORD=supersecret alpine sleep 3600

# Inspect the container — password visible
docker inspect leaky | grep DB_PASSWORD
# Output: "DB_PASSWORD=supersecret"

# Or read from /proc
docker exec leaky cat /proc/1/environ | tr '\0' '\n' | grep DB_PASSWORD
# Output: DB_PASSWORD=supersecret

Docker Swarm Secrets: The Built-In Solution

Docker Swarm has native secret management. Secrets are encrypted at rest in the Swarm store and only decrypted when a container that needs them is scheduled. They're mounted as files in /run/secrets/ by default, with in-memory tmpfs. No disk writes. No env var leaks. The flow: docker secret create stores the secret, then you grant access to services via --secret. The container reads the file at startup. Rotation requires updating the secret and redeploying the service. For zero-downtime rotation, you can mount multiple versions and have the app watch for changes.

// io.thecodeforge — DevOps tutorial

# Create a secret from stdin
echo "my-db-password" | docker secret create db_password -

# Verify
docker secret ls

# Deploy a service with the secret
docker service create --name web \
  --secret db_password \
  --publish 80:80 \
  nginx:alpine

# Inside the container, the secret is at /run/secrets/db_password
docker exec $(docker ps -q) cat /run/secrets/db_password
# Output: my-db-password

Kubernetes Secrets: More Power, More Pitfalls

Kubernetes Secrets are similar but more flexible — and more dangerous. By default, Secrets are stored unencrypted in etcd. You must enable encryption at rest. They can be mounted as volumes or injected as env vars (don't do the latter). The real power is in integration with external vaults like HashiCorp Vault via CSI drivers or sidecars. For production, never use plain Kubernetes Secrets for anything sensitive. Always enable encryption and use RBAC to restrict access. The classic mistake: storing a cloud API key in a Secret and then accidentally committing the YAML to git.

// io.thecodeforge — DevOps tutorial

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
type: Opaque
data:
  password: bXktZGItcGFzc3dvcmQ=  # base64 of 'my-db-password'
---
apiVersion: v1
kind: Pod
metadata:
  name: web
spec:
  containers:
  - name: web
    image: nginx:alpine
    volumeMounts:
    - name: secrets
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: db-credentials

Secret Rotation Without Downtime

Rotating secrets is where most teams fail. The naive approach: update the secret, restart all containers. That causes downtime. Better: mount the secret as a file and have the application watch for file changes (e.g., inotify). When the file changes, reload the connection. For databases, use a connection pool that supports credential refresh (like HikariCP with a DataSource that reads from a file). In Swarm, you can update a secret and redeploy the service with --update-delay for rolling updates. In Kubernetes, use a controller that watches the Secret and triggers a rolling restart via annotation change.

// io.thecodeforge — DevOps tutorial

import os
import time
import signal

SECRET_PATH = '/run/secrets/db_password'

def load_secret():
    with open(SECRET_PATH) as f:
        return f.read().strip()

def reload_handler(signum, frame):
    global db_password
    db_password = load_secret()
    print(f"Secret reloaded at {time.ctime()}")

# Register SIGHUP handler for manual reload
signal.signal(signal.SIGHUP, reload_handler)

# Initial load
db_password = load_secret()
print(f"Initial password: {db_password}")

# Simulate main loop
while True:
    time.sleep(10)
    # In production, use inotify or watchdog library for file change detection

External Vault Integration: HashiCorp Vault with Docker

For serious production environments, built-in secrets managers are too limited. You need audit logs, dynamic secrets (short-lived credentials), and fine-grained access control. HashiCorp Vault is the gold standard. Integrate with Docker via the Vault agent sidecar pattern: a container that authenticates to Vault, retrieves secrets, and writes them to a shared volume. The application reads from that volume. Vault handles rotation automatically. The downside: complexity. You now manage Vault itself. But for compliance-heavy environments (PCI-DSS, HIPAA), it's non-negotiable.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  vault-agent:
    image: vault:latest
    command: agent -config=/etc/vault/config.hcl
    volumes:
      - shared-secrets:/secrets
      - ./vault-config:/etc/vault
    environment:
      VAULT_ADDR: https://vault.example.com
  app:
    image: myapp:latest
    volumes:
      - shared-secrets:/secrets:ro
    depends_on:
      - vault-agent
volumes:
  shared-secrets:

When Not to Use Docker Secrets

Docker secrets are overkill for local development. Use .env files or docker-compose's env_file — just don't commit them to git. For single-node deployments, bind-mounting a secrets directory with restricted permissions is simpler and equally secure. Also, if your orchestrator doesn't support secrets (e.g., plain Docker Compose without Swarm), you're stuck with workarounds. In those cases, use a vault sidecar or encrypt the env var with a key that the app decrypts at startup. The key itself must be stored securely — a chicken-and-egg problem that vault solves.

// io.thecodeforge — DevOps tutorial

# For local dev, just mount a file with restricted permissions
mkdir -p ./secrets
echo "dev-password" > ./secrets/db_password
chmod 0400 ./secrets/db_password
chown 1000:1000 ./secrets/db_password

docker run -v $(pwd)/secrets:/run/secrets:ro alpine cat /run/secrets/db_password
# Output: dev-password