Docker Swarm — Why 4 Managers Caused a 3-Hour Outage

Every production app eventually outgrows a single server. Traffic spikes, hardware fails, deployments need to happen without downtime. Docker Swarm is the native clustering and orchestration layer baked directly into the Docker Engine.

Swarm solves coordination across multiple hosts. When you have ten nodes, you need something to decide where a container lands, what happens when a node dies, how containers on different hosts communicate, and how you push a new image without dropping requests. Swarm encodes those answers into a distributed state machine backed by the Raft consensus algorithm.

Common misconceptions: Swarm is not deprecated (Docker continues to maintain it alongside Compose). Swarm is not Kubernetes-lite (it has a fundamentally different architecture — no pods, no CRDs, no etcd). Swarm's simplicity is its strength for small-to-medium deployments that do not need Kubernetes' complexity.

Raft Consensus and Manager Node Architecture

Swarm's cluster state is stored in a distributed log managed by the Raft consensus algorithm. Every manager node runs a full copy of the Raft log. State changes (service updates, node joins, secret creation) are proposed by the leader, replicated to a quorum of followers, and then committed.

The quorum formula is floor(n/2) + 1, where n is the number of managers. With 3 managers, quorum is 2. With 5 managers, quorum is 3. The cluster can tolerate floor((n-1)/2) manager failures. With 3 managers, you can lose 1. With 5 managers, you can lose 2.

An even number of managers provides no additional fault tolerance over the next lower odd number. With 4 managers, quorum is 3 — you can still only lose 1 manager, same as with 3 managers. The 4th node is wasted.

Leader election: When the leader fails or becomes unreachable, the remaining managers hold an election. The manager with the most up-to-date Raft log and the lowest election timeout wins. The default election timeout is 1 second. Network partitions can cause split-brain if two groups of managers each elect their own leader, but only the group with quorum can commit new state changes.

Failure scenario — manager resource starvation: A team ran a memory-intensive batch job on a manager node. The job consumed all available RAM, causing the Docker daemon to be OOM-killed. The daemon restart triggered a Raft leader election. During the election window (1-2 seconds), no state changes could be committed. The team noticed brief delays in service updates. The fix: cordon manager nodes from workloads using docker node update --availability drain <manager-node>.

#!/bin/bash
# Swarm cluster bootstrap with proper manager configuration

# ── Initialize the Swarm on the first manager ─────────────────────
# --advertise-addr: the IP other nodes will use to reach this manager
# --listen-addr: the interface the manager binds to
docker swarm init \
  --advertise-addr 10.0.1.10 \
  --listen-addr 0.0.0.0:2377 \
  --data-path-addr 10.0.1.10

# Get the join tokens
docker swarm join-token manager  # For other managers
docker swarm join-token worker   # For workers

# ── Join additional managers (run on each new manager node) ────────
docker swarm join \
  --token SWMTKN-1-xxxxx-manager-token-xxxxx \
  --advertise-addr 10.0.1.11 \
  10.0.1.10:2377

# ── Verify manager count (should be 3 or 5, never even) ──────────
docker node ls --filter role=manager
# ID    HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
# abc * manager-1  Ready     Active         Leader           24.0.7
# def   manager-2  Ready     Active         Reachable        24.0.7
# ghi   manager-3  Ready     Active         Reachable        24.0.7

# ── Drain manager nodes to prevent workloads from running on them ─
for node in $(docker node ls --filter role=manager -q); do
  docker node update --availability drain $node
done
# Drained managers cannot run tasks — they are dedicated to orchestration

# ── Check Raft cluster health ─────────────────────────────────────
docker info --format '{{.Swarm.Nodes}} managers, {{.Swarm.Nodes}} total'
# Or inspect the Raft status on each manager
docker node inspect self --format '{{.ManagerStatus.Leader}}'
# true on the leader, false on followers

# ── Configure auto-lock (encrypt Raft logs at rest) ───────────────
docker swarm update --autolock=true
# This requires unlocking the swarm after daemon restart:
# docker swarm unlock
# Enter unlock key: SWMKEY-1-xxxxx

Service Scheduling, Placement Constraints and Resource Limits

A Swarm service is a declarative specification of the desired state: which image to run, how many replicas, resource limits, placement constraints, and update policy. The Swarm scheduler assigns tasks (individual containers) to nodes that satisfy the constraints and have available resources.

Scheduling algorithm: Swarm uses a spread scheduler by default — it places tasks on the node with the fewest existing tasks of the same service. This provides natural load distribution. You can override this with placement constraints and preferences.

Placement constraints: Hard requirements that a node must satisfy. Examples: - node.role==manager: only run on manager nodes - node.labels.zone==us-east-1a: only run in a specific availability zone - node.hostname==worker-3: pin to a specific node

Placement preferences: Soft preferences that guide scheduling but do not prevent placement. Example: --placement-pref 'spread=node.labels.zone' distributes tasks evenly across zones.

Resource limits: - --limit-cpu: maximum CPU a task can consume (e.g., 0.5 = half a core) - --limit-memory: maximum memory (e.g., 512m) - --reserve-cpu: guaranteed CPU allocation - --reserve-memory: guaranteed memory allocation

Without resource limits, a single misbehaving container can consume all resources on a node, starving other tasks. Resource reservations ensure critical services always have the resources they need.

Failure scenario — no resource limits, noisy neighbor: A team deployed a memory-intensive analytics service without --limit-memory. The service gradually consumed all available RAM on a worker node. The kernel OOM-killed other containers on the same node, including a critical payment service. The payment service was rescheduled to another node (Swarm's self-healing), but the 30-second rescheduling delay caused a brief payment outage. The fix: add --limit-memory to all services and --reserve-memory for critical services.

#!/bin/bash
# Production-grade service deployment with constraints and resource limits

# ── Deploy a service with full production settings ────────────────
docker service create \
  --name io-thecodeforge-api \
  --image io.thecodeforge/api:v2.3.1 \
  --replicas 6 \
  \
  # Resource limits — hard ceiling
  --limit-cpu 1.0 \
  --limit-memory 512m \
  \
  # Resource reservations — guaranteed allocation
  --reserve-cpu 0.25 \
  --reserve-memory 128m \
  \
  # Placement: spread across availability zones
  --placement-pref 'spread=node.labels.zone' \
  \
  # Constraint: never run on manager nodes
  --constraint 'node.role!=manager' \
  \
  # Constraint: only run on nodes with SSD label
  --constraint 'node.labels.disk==ssd' \
  \
  # Health check
  --health-cmd 'curl -f http://localhost:8080/health || exit 1' \
  --health-interval 10s \
  --health-timeout 5s \
  --health-retries 3 \
  --health-start-period 30s \
  \
  # Rolling update policy
  --update-parallelism 2 \
  --update-delay 10s \
  --update-failure-action rollback \
  --update-order start-first \
  \
  # Rollback policy
  --rollback-parallelism 1 \
  --rollback-delay 5s \
  \
  # Network
  --network io-thecodeforge-overlay \
  --publish published=8080,target=8080 \
  \
  # Environment
  --env DATABASE_URL='{{DATABASE_URL}}' \
  --secret io-thecodeforge-db-password \
  \
  # Restart policy
  --restart-condition on-failure \
  --restart-delay 5s \
  --restart-max-attempts 3 \
  --restart-window 60s

# ── Label nodes for placement constraints ─────────────────────────
docker node update --label-add zone=us-east-1a worker-1
docker node update --label-add zone=us-east-1b worker-2
docker node update --label-add zone=us-east-1a worker-3
docker node update --label-add disk=ssd worker-1
docker node update --label-add disk=ssd worker-2

# ── Verify placement ──────────────────────────────────────────────
docker service ps io-thecodeforge-api --format '{{.Name}} {{.Node}} {{.CurrentState}}'
# api.1  worker-1  Running
# api.2  worker-2  Running
# api.3  worker-1  Running
# api.4  worker-2  Running
# api.5  worker-3  Running
# api.6  worker-3  Running

Overlay Networks and Cross-Host Container Communication

Docker Swarm uses overlay networks to enable containers on different hosts to communicate as if they were on the same network. The overlay network uses VXLAN (Virtual Extensible LAN) encapsulation to tunnel Layer 2 traffic over the underlying Layer 3 network.

How it works: When container A on node 1 sends a packet to container B on node 2, the VXLAN driver encapsulates the packet in a UDP datagram on port 4789 and sends it to node 2. Node 2 decapsulates the packet and delivers it to container B. The containers see each other's overlay IP addresses as if they were on the same LAN.

The ingress routing mesh: When you publish a port with --publish, Swarm creates a route in the ingress network that load-balances incoming traffic across all nodes running the service. Any node in the cluster can receive traffic for any service, regardless of whether that node is running the service's containers. The routing mesh forwards the traffic to a node that is running a healthy task.

The extra-hop problem: The routing mesh adds one network hop. A request to node 1 may be routed to a container on node 3. This adds latency. For latency-sensitive services, use host-mode publishing: --publish published=8080,target=8080,mode=host. This bypasses the routing mesh and binds directly to the host's port. The trade-off: only nodes running the service's containers accept traffic — you lose the any-node routing benefit.

Failure scenario — VXLAN port blocked by firewall: A team deployed a 3-node Swarm cluster across two data centers. Containers in data center A could not reach containers in data center B. The team spent 4 hours debugging DNS, service discovery, and overlay configuration. The root cause: the firewall between data centers blocked UDP port 4789 (VXLAN). After opening the port, overlay connectivity was restored immediately.

#!/bin/bash
# Overlay network setup and troubleshooting

# ── Create an overlay network with encryption ─────────────────────
docker network create \
  --driver overlay \
  --attachable \
  --opt encrypted \
  --subnet 10.0.10.0/24 \
  io-thecodeforge-overlay
# --driver overlay: VXLAN-based cross-host networking
# --attachable: allows standalone containers to join (useful for debugging)
# --opt encrypted: encrypts VXLAN traffic with IPsec (adds ~10% overhead)
# --subnet: explicit IP range for the overlay network

# ── Deploy a service on the overlay network ───────────────────────
docker service create \
  --name api \
  --network io-thecodeforge-overlay \
  --replicas 3 \
  io.thecodeforge/api:v2.3.1

# ── Verify overlay network peers (should list all nodes) ──────────
docker network inspect io-thecodeforge-overlay --format '{{json .Peers}}' | python3 -m json.tool
# Each peer represents a node participating in the overlay network
# If a peer is missing, that node cannot communicate on the overlay

# ── Test cross-host connectivity ──────────────────────────────────
# From any node, run a debug container on the overlay network
docker run --rm -it --network io-thecodeforge-overlay alpine sh
# Inside the container:
# ping <overlay-ip-of-service-task>
# nslookup tasks.api  # DNS round-robin for all service tasks

# ── Required ports for Swarm networking ───────────────────────────
# TCP 2377: Swarm cluster management (Raft)
# TCP/UDP 7946: Gossip-based node discovery
# UDP 4789: VXLAN overlay network traffic
# Protocol 50 (ESP): IPsec encryption (if --opt encrypted)

# ── Host-mode publishing (bypass routing mesh) ────────────────────
docker service create \
  --name api-latency-sensitive \
  --network io-thecodeforge-overlay \
  --publish published=8080,target=8080,mode=host \
  --mode global \
  io.thecodeforge/api:v2.3.1
# mode=global: one task per node (every node runs the service)
# mode=host: binds directly to host port 8080, no routing mesh hop

Rolling Updates, Rollback and Zero-Downtime Deployments

Swarm's rolling update mechanism replaces old containers with new ones incrementally, ensuring the service remains available throughout the deployment. The update configuration controls the pace and failure behavior.

Update parameters: - --update-parallelism: how many tasks to update simultaneously (default: 1) - --update-delay: wait time between updating batches (default: 0s) - --update-failure-action: what to do if a new task fails (pause, continue, rollback) - --update-order: start-first (new container starts before old stops) or stop-first (old stops before new starts) - --update-max-failure-ratio: percentage of failures that triggers the failure action

The start-first vs stop-first trade-off: - start-first: zero downtime, but temporarily doubles resource usage during deployment - stop-first: lower resource usage, but brief window where one fewer replica is running

Rollback: If a rolling update fails, Swarm can automatically roll back to the previous version. The rollback configuration mirrors the update configuration. Manual rollback: docker service rollback <service>.

Failure scenario — update without health check causes cascading failure: A team deployed a new API version with a startup bug that caused the health check to fail after 30 seconds. The team did not configure --health-start-period. The health check failed immediately (before the app was ready), causing Swarm to mark the task as failed. With --update-failure-action continue (the default), Swarm continued replacing all healthy containers with the failing new version. Within 2 minutes, all containers were running the broken version. The fix: set --update-failure-action rollback and configure --health-start-period to allow startup time.

#!/bin/bash
# Zero-downtime rolling update with automatic rollback

# ── Initial deployment ────────────────────────────────────────────
docker service create \
  --name io-thecodeforge-api \
  --image io.thecodeforge/api:v2.3.0 \
  --replicas 6 \
  --limit-cpu 1.0 \
  --limit-memory 512m \
  --health-cmd 'curl -f http://localhost:8080/health || exit 1' \
  --health-interval 10s \
  --health-timeout 5s \
  --health-retries 3 \
  --health-start-period 40s \
  \
  # Rolling update: 2 at a time, 10s delay, auto-rollback on failure
  --update-parallelism 2 \
  --update-delay 10s \
  --update-failure-action rollback \
  --update-max-failure-ratio 0.25 \
  --update-order start-first \
  \
  # Rollback policy
  --rollback-parallelism 1 \
  --rollback-delay 5s \
  --rollback-order stop-first \
  \
  --network io-thecodeforge-overlay \
  --publish published=8080,target=8080 \
  io.thecodeforge/api:v2.3.0

# ── Rolling update to new version ─────────────────────────────────
docker service update \
  --image io.thecodeforge/api:v2.3.1 \
  --update-parallelism 2 \
  --update-delay 10s \
  io-thecodeforge-api

# ── Monitor the update progress ───────────────────────────────────
docker service ps io-thecodeforge-api \
  --format '{{.Name}} {{.Image}} {{.CurrentState}} {{.Error}}' \
  | head -20
# You will see old tasks shutting down and new tasks starting

# ── Manual rollback if needed ─────────────────────────────────────
docker service rollback io-thecodeforge-api
# Reverts to the previous image and configuration

# ── Force update (redeploy without changing image) ────────────────
docker service update --force io-thecodeforge-api
# Useful when container config has changed but image tag is the same

Swarm Secrets and Configs — Immutable, Encrypted, Rotatable

Docker Swarm provides built-in secrets management through the Raft log. Secrets are encrypted at rest and in transit, mounted as files in /run/secrets/ inside containers, and never written to image layers.

How secrets work: - docker secret create: stores the secret in the Raft log (encrypted with the swarm unlock key) - The secret is distributed to every manager node (encrypted) - When a service references a secret, it is mounted as a file at /run/secrets/<secret-name> - Secrets are immutable — updating a secret creates a new version

How configs work: - docker config create: stores configuration files in the Raft log - Configs are mounted as files in the container (not encrypted at rest — use secrets for sensitive data) - Useful for nginx.conf, application.yaml, or any configuration file

Secret rotation: Secrets are immutable. To rotate a secret: 1. Create a new secret: docker secret create db-password-v2 - 2. Update the service to use the new secret: docker service update --secret-rm db-password --secret-add db-password-v2 <service> 3. The service restarts with the new secret mounted 4. Delete the old secret: docker secret rm db-password

Failure scenario — secret not updating in running service: A team updated a database password by creating a new secret and updating the service. However, the application inside the container still read the old password from /run/secrets/db-password. The team did not realize that Docker secrets are immutable — the old secret file remained mounted until the service was explicitly updated to remove it. The fix: use --secret-rm to remove the old secret and --secret-add to add the new one in the same update command.

#!/bin/bash
# Secrets management in Docker Swarm

# ── Create a secret from stdin ────────────────────────────────────
echo 's3cret_p@ssw0rd' | docker secret create io-thecodeforge-db-password -

# ── Create a secret from a file ───────────────────────────────────
docker secret create io-thecodeforge-tls-cert /path/to/cert.pem

# ── Create a config (non-sensitive configuration) ─────────────────
docker config create io-thecodeforge-nginx-conf /path/to/nginx.conf

# ── Deploy a service with secrets and configs ─────────────────────
docker service create \
  --name io-thecodeforge-api \
  --secret io-thecodeforge-db-password \
  --secret io-thecodeforge-tls-cert \
  --config source=io-thecodeforge-nginx-conf,target=/etc/nginx/nginx.conf \
  io.thecodeforge/api:v2.3.1

# ── Access secrets inside the container ────────────────────────────
docker exec <container> cat /run/secrets/io-thecodeforge-db-password
# Output: s3cret_p@ssw0rd

docker exec <container> ls /run/secrets/
# io-thecodeforge-db-password
# io-thecodeforge-tls-cert

# ── Rotate a secret ───────────────────────────────────────────────
# Step 1: Create new version
echo 'new_s3cret_p@ssw0rd' | docker secret create io-thecodeforge-db-password-v2 -

# Step 2: Update service — remove old, add new
docker service update \
  --secret-rm io-thecodeforge-db-password \
  --secret-add io-thecodeforge-db-password-v2 \
  io-thecodeforge-api

# Step 3: Verify new secret is mounted
docker exec <container> cat /run/secrets/io-thecodeforge-db-password-v2
# Output: new_s3cret_p@ssw0rd

# Step 4: Clean up old secret
docker secret rm io-thecodeforge-db-password

# ── List all secrets ──────────────────────────────────────────────
docker secret ls
# ID          NAME                          CREATED
# abc123      io-thecodeforge-db-password   2 hours ago
def456      io-thecodeforge-tls-cert      2 hours ago