Docker Resource Limits and cgroups: Stop Your Containers From Eating the Host

You've seen it happen: a memory leak in a container brings down the entire host. Or a CPU-bound process throttles your database container into submission. Docker resource limits exist to prevent exactly this chaos. But most developers slap on a --memory=512m and call it a day, not understanding that cgroups are a deep kernel feature with sharp edges. This article gives you the internals, the gotchas, and the production patterns that keep your containers from eating the host alive. By the end, you'll know exactly how to set limits, debug cgroup issues, and design resource isolation for real workloads.

What cgroups Actually Do Under the Hood

cgroups (control groups) are a Linux kernel feature that limits, accounts for, and isolates resource usage of process groups. Docker uses cgroups v1 (legacy) or v2 (newer) to enforce limits. Each container gets its own cgroup hierarchy under /sys/fs/cgroup/. When you set --memory=512m, Docker writes to memory.limit_in_bytes. The kernel enforces this at allocation time: if a process tries to allocate beyond the limit, it gets OOM-killed or throttled. CPU limits use Completely Fair Scheduler (CFS) quotas: cpu.cfs_period_us and cpu.cfs_quota_us. For example, --cpus=1.5 sets quota to 150000 in a 100000 period, meaning the container gets 1.5 cores worth of CPU time per period. Blkio limits throttle disk I/O using token bucket algorithms. Understanding these files lets you debug limits directly without Docker.

// io.thecodeforge — DevOps tutorial

# Check memory limit for a running container
CONTAINER_ID=$(docker run -d --memory=512m alpine sleep 3600)

# Find the cgroup path (cgroups v1 example)
CGROUP_PATH=$(docker inspect --format '{{.HostConfig.CgroupParent}}' $CONTAINER_ID)
# On modern systems, it's often under /sys/fs/cgroup/memory/docker/<container-id>

# Read the memory limit
cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.limit_in_bytes
# Output: 536870912 (512MB in bytes)

# Read current memory usage
cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.usage_in_bytes

# For cgroups v2, use:
cat /sys/fs/cgroup/system.slice/docker-$CONTAINER_ID.scope/memory.max

Memory Limits: The Silent Killer

Memory limits are the most critical resource limit. Without them, a container can exhaust host memory, triggering the kernel OOM killer which may kill unrelated processes. Docker's --memory sets the hard limit. But there's a twist: --memory-swap defaults to twice the memory limit, meaning the container can use swap up to that total. This can mask memory leaks and cause performance degradation. Always set --memory-swap equal to --memory to disable swap. Also consider --memory-reservation — a soft limit that the kernel tries to enforce but can exceed under pressure. Use it for overcommit scenarios. Production gotcha: Java JVM doesn't respect cgroup memory limits by default in older versions. Use -XX:+UseContainerSupport and -XX:MaxRAMPercentage=75.0 to make it play nice.

// io.thecodeforge — DevOps tutorial

# Run a container with 512MB memory, no swap
# --memory-swap equal to --memory disables swap
docker run -d --name memory-test \
  --memory=512m \
  --memory-swap=512m \
  --memory-reservation=256m \
  alpine sleep 3600

# Simulate memory pressure inside container
docker exec memory-test sh -c "
  # Allocate 600MB (will be killed)
  dd if=/dev/zero of=/dev/null bs=1M count=600 &
  sleep 2
  # Check if process is still alive
  jobs
"
# Expected: process gets killed (exit code 137) because allocation exceeds limit

# Check OOM kills
cat /sys/fs/cgroup/memory/docker/$(docker ps -q --filter name=memory-test)/memory.oom_control
# Output: oom_kill_disable 0  under_oom 1  oom_kill 1

CPU Limits: Not All Cores Are Equal

CPU limits are trickier than they look. --cpus sets a hard cap: a container with --cpus=1.5 gets 1.5 CPU-seconds per second. But this is enforced via CFS quotas, which can cause throttling even when the host is idle. The kernel uses a 100ms period by default. If your container bursts CPU for 50ms then sleeps, it's fine. But if it uses 150ms continuously, it gets throttled for 50ms. This can hurt latency-sensitive apps. For proportional sharing (no hard cap), use --cpu-shares (default 1024). A container with shares=2048 gets twice as much CPU as one with 1024 when there's contention. But if the host is idle, it can use all cores. Use --cpus for predictable limits, --cpu-shares for fair sharing. Production gotcha: --cpus doesn't pin to specific cores. Use --cpuset-cpus for NUMA-aware pinning.

// io.thecodeforge — DevOps tutorial

# Run two containers: one with 1 CPU, one with 2 CPUs
# Both will stress CPU

docker run -d --name cpu-limited --cpus=1 alpine sh -c "
  while true; do :; done
"

docker run -d --name cpu-unlimited alpine sh -c "
  while true; do :; done
"

# Check CPU usage with docker stats
docker stats --no-stream cpu-limited cpu-unlimited
# Expected: cpu-limited ~100%, cpu-unlimited ~100% (if host has 2+ cores)
# But if host is single-core, cpu-unlimited will get ~100% and cpu-limited will be throttled

# Check throttling stats inside cpu-limited
cat /sys/fs/cgroup/cpu/docker/$(docker ps -q --filter name=cpu-limited)/cpu.stat
# Output: nr_periods 100  nr_throttled 50  throttled_time 5000000

Blkio Limits: Taming Disk I/O

Disk I/O limits are often overlooked until a batch job saturates the disk and your database latency spikes. Docker uses --blkio-weight for proportional I/O scheduling (similar to CPU shares). Range is 10-1000, default 500. A container with weight 1000 gets twice the I/O of one with 500 under contention. For hard limits, use --device-read-bps and --device-write-bps to cap throughput, or --device-read-iops and --device-write-iops for IOPS limits. These use the kernel's blkio cgroup controller. Note: blkio limits only apply to direct I/O, not buffered writes. Buffered writes are accounted to the page cache, which is shared. Production gotcha: On cgroups v1, blkio limits don't work with CFQ scheduler (deprecated). Use none scheduler or switch to v2.

// io.thecodeforge — DevOps tutorial

# Limit write throughput to 10MB/s on /dev/sda
docker run -d --name io-limited \
  --device-write-bps /dev/sda:10mb \
  alpine sh -c "
    dd if=/dev/zero of=/tmp/test bs=1M count=100 oflag=direct
  "

# Monitor write speed inside container
docker exec io-limited sh -c "
  dd if=/dev/zero of=/tmp/test2 bs=1M count=100 oflag=direct status=progress
"
# Expected: speed ~10MB/s

# Check blkio stats
cat /sys/fs/cgroup/blkio/docker/$(docker ps -q --filter name=io-limited)/blkio.throttle.io_service_bytes
# Output: 8:0 Write 10485760 (10MB)

cgroups v1 vs v2: What Changed and Why It Matters

cgroups v2 unified the hierarchy and fixed many v1 inconsistencies. Docker 20.10+ supports v2 on Linux 4.15+ with systemd. Key differences: v2 has a single hierarchy per controller, no more multiple mounts. Memory and CPU are in the same tree. The memory.limit_in_bytes becomes memory.max. CPU quota files are under cpu.max instead of cpu.cfs_quota_us. Blkio is replaced by io controller with io.max for limits. Docker abstracts most of this, but when debugging, you need to know which version your host uses. Check with stat -fc %T /sys/fs/cgroup/. If it says cgroup2fs, you're on v2. Production gotcha: Some tools like lxcfs or older monitoring agents may not work with v2. Also, swap accounting is disabled by default in v2; enable with swapaccount=1 kernel parameter.

// io.thecodeforge — DevOps tutorial

# Check cgroup version
stat -fc %T /sys/fs/cgroup/
# Output: cgroup2fs (v2) or tmpfs (v1)

# For v2, check memory limit
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/memory.max

# For v1, check memory limit
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes

# Check if swap accounting is enabled (v2)
cat /sys/fs/cgroup/memory.swap.current
# If file doesn't exist, swap accounting is off

Resource Limits in Docker Compose and Kubernetes

Docker Compose uses the same syntax as docker run under deploy.resources. For example: ``yaml services: app: image: myapp deploy: resources: limits: cpus: '0.5' memory: 256M reservations: cpus: '0.25' memory: 128M ` In Kubernetes, resource limits are set in pod specs. But Kubernetes uses its own cgroup management (via kubelet). Docker's limits are ignored when running under Kubernetes. Instead, set resources.limits in the pod spec. Important: Kubernetes enforces limits via cgroups, but it also uses QoS classes (Guaranteed, Burstable, BestEffort) based on whether limits equal requests. For guaranteed QoS, set limits == requests`. Production gotcha: If you set CPU limits in Kubernetes, your container may be throttled even if the node has spare CPU. This is due to CFS quotas. Consider using CPU manager policies for latency-sensitive apps.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 128M
        reservations:
          cpus: '0.25'
          memory: 64M
    # Note: deploy.resources only works in swarm mode, not docker-compose up
    # For docker-compose up, use:
    # mem_limit: 128m
    # cpus: 0.5

When Not to Use Resource Limits

Resource limits aren't always the answer. For single-container hosts or dedicated instances, limits add overhead and complexity. If your container is the only workload, skip limits and use host-level monitoring instead. Also, for batch jobs that need maximum throughput, limits can cause unnecessary throttling. Use --cpu-shares instead of --cpus to allow bursting. For memory, if your app has predictable usage, set a reservation but no hard limit — let the kernel OOM killer handle extreme cases. But this is risky. My rule: always set memory limits (hard or soft) for any container in a multi-tenant host. CPU limits are optional for low-priority workloads.