Docker Resource Limits and cgroups: Stop Your Containers From Eating the Host

You've seen it happen: a memory leak in a container brings down the entire host. Or a CPU-bound process throttles your database container into submission. Docker resource limits exist to prevent exactly this chaos. But most developers slap on a --memory=512m and call it a day, not understanding that cgroups are a deep kernel feature with sharp edges. This article gives you the internals, the gotchas, and the production patterns that keep your containers from eating the host alive. By the end, you'll know exactly how to set limits, debug cgroup issues, and design resource isolation for real workloads.

What cgroups Actually Do Under the Hood

cgroups (control groups) are a Linux kernel feature that limits, accounts for, and isolates resource usage of process groups. Docker uses cgroups v1 (legacy) or v2 (newer) to enforce limits. Each container gets its own cgroup hierarchy under /sys/fs/cgroup/. When you set --memory=512m, Docker writes to memory.limit_in_bytes. The kernel enforces this at allocation time: if a process tries to allocate beyond the limit, it gets OOM-killed or throttled. CPU limits use Completely Fair Scheduler (CFS) quotas: cpu.cfs_period_us and cpu.cfs_quota_us. For example, --cpus=1.5 sets quota to 150000 in a 100000 period, meaning the container gets 1.5 cores worth of CPU time per period. Blkio limits throttle disk I/O using token bucket algorithms. Understanding these files lets you debug limits directly without Docker.

// io.thecodeforge — DevOps tutorial

# Check memory limit for a running container
CONTAINER_ID=$(docker run -d --memory=512m alpine sleep 3600)

# Find the cgroup path (cgroups v1 example)
CGROUP_PATH=$(docker inspect --format '{{.HostConfig.CgroupParent}}' $CONTAINER_ID)
# On modern systems, it's often under /sys/fs/cgroup/memory/docker/<container-id>

# Read the memory limit
cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.limit_in_bytes
# Output: 536870912 (512MB in bytes)

# Read current memory usage
cat /sys/fs/cgroup/memory/docker/$CONTAINER_ID/memory.usage_in_bytes

# For cgroups v2, use:
cat /sys/fs/cgroup/system.slice/docker-$CONTAINER_ID.scope/memory.max

Memory Limits: The Silent Killer

Memory limits are the most critical resource limit. Without them, a container can exhaust host memory, triggering the kernel OOM killer which may kill unrelated processes. Docker's --memory sets the hard limit. But there's a twist: --memory-swap defaults to twice the memory limit, meaning the container can use swap up to that total. This can mask memory leaks and cause performance degradation. Always set --memory-swap equal to --memory to disable swap. Also consider --memory-reservation — a soft limit that the kernel tries to enforce but can exceed under pressure. Use it for overcommit scenarios. Production gotcha: Java JVM doesn't respect cgroup memory limits by default in older versions. Use -XX:+UseContainerSupport and -XX:MaxRAMPercentage=75.0 to make it play nice.

// io.thecodeforge — DevOps tutorial

# Run a container with 512MB memory, no swap
# --memory-swap equal to --memory disables swap
docker run -d --name memory-test \
  --memory=512m \
  --memory-swap=512m \
  --memory-reservation=256m \
  alpine sleep 3600

# Simulate memory pressure inside container
docker exec memory-test sh -c "
  # Allocate 600MB (will be killed)
  dd if=/dev/zero of=/dev/null bs=1M count=600 &
  sleep 2
  # Check if process is still alive
  jobs
"
# Expected: process gets killed (exit code 137) because allocation exceeds limit

# Check OOM kills
cat /sys/fs/cgroup/memory/docker/$(docker ps -q --filter name=memory-test)/memory.oom_control
# Output: oom_kill_disable 0  under_oom 1  oom_kill 1

CPU Limits: Not All Cores Are Equal

CPU limits are trickier than they look. --cpus sets a hard cap: a container with --cpus=1.5 gets 1.5 CPU-seconds per second. But this is enforced via CFS quotas, which can cause throttling even when the host is idle. The kernel uses a 100ms period by default. If your container bursts CPU for 50ms then sleeps, it's fine. But if it uses 150ms continuously, it gets throttled for 50ms. This can hurt latency-sensitive apps. For proportional sharing (no hard cap), use --cpu-shares (default 1024). A container with shares=2048 gets twice as much CPU as one with 1024 when there's contention. But if the host is idle, it can use all cores. Use --cpus for predictable limits, --cpu-shares for fair sharing. Production gotcha: --cpus doesn't pin to specific cores. Use --cpuset-cpus for NUMA-aware pinning.

// io.thecodeforge — DevOps tutorial

# Run two containers: one with 1 CPU, one with 2 CPUs
# Both will stress CPU

docker run -d --name cpu-limited --cpus=1 alpine sh -c "
  while true; do :; done
"

docker run -d --name cpu-unlimited alpine sh -c "
  while true; do :; done
"

# Check CPU usage with docker stats
docker stats --no-stream cpu-limited cpu-unlimited
# Expected: cpu-limited ~100%, cpu-unlimited ~100% (if host has 2+ cores)
# But if host is single-core, cpu-unlimited will get ~100% and cpu-limited will be throttled

# Check throttling stats inside cpu-limited
cat /sys/fs/cgroup/cpu/docker/$(docker ps -q --filter name=cpu-limited)/cpu.stat
# Output: nr_periods 100  nr_throttled 50  throttled_time 5000000

Blkio Limits: Taming Disk I/O

Disk I/O limits are often overlooked until a batch job saturates the disk and your database latency spikes. Docker uses --blkio-weight for proportional I/O scheduling (similar to CPU shares). Range is 10-1000, default 500. A container with weight 1000 gets twice the I/O of one with 500 under contention. For hard limits, use --device-read-bps and --device-write-bps to cap throughput, or --device-read-iops and --device-write-iops for IOPS limits. These use the kernel's blkio cgroup controller. Note: blkio limits only apply to direct I/O, not buffered writes. Buffered writes are accounted to the page cache, which is shared. Production gotcha: On cgroups v1, blkio limits don't work with CFQ scheduler (deprecated). Use none scheduler or switch to v2.

// io.thecodeforge — DevOps tutorial

# Limit write throughput to 10MB/s on /dev/sda
docker run -d --name io-limited \
  --device-write-bps /dev/sda:10mb \
  alpine sh -c "
    dd if=/dev/zero of=/tmp/test bs=1M count=100 oflag=direct
  "

# Monitor write speed inside container
docker exec io-limited sh -c "
  dd if=/dev/zero of=/tmp/test2 bs=1M count=100 oflag=direct status=progress
"
# Expected: speed ~10MB/s

# Check blkio stats
cat /sys/fs/cgroup/blkio/docker/$(docker ps -q --filter name=io-limited)/blkio.throttle.io_service_bytes
# Output: 8:0 Write 10485760 (10MB)

cgroups v1 vs v2: What Changed and Why It Matters

cgroups v2 unified the hierarchy and fixed many v1 inconsistencies. Docker 20.10+ supports v2 on Linux 4.15+ with systemd. Key differences: v2 has a single hierarchy per controller, no more multiple mounts. Memory and CPU are in the same tree. The memory.limit_in_bytes becomes memory.max. CPU quota files are under cpu.max instead of cpu.cfs_quota_us. Blkio is replaced by io controller with io.max for limits. Docker abstracts most of this, but when debugging, you need to know which version your host uses. Check with stat -fc %T /sys/fs/cgroup/. If it says cgroup2fs, you're on v2. Production gotcha: Some tools like lxcfs or older monitoring agents may not work with v2. Also, swap accounting is disabled by default in v2; enable with swapaccount=1 kernel parameter.

// io.thecodeforge — DevOps tutorial

# Check cgroup version
stat -fc %T /sys/fs/cgroup/
# Output: cgroup2fs (v2) or tmpfs (v1)

# For v2, check memory limit
cat /sys/fs/cgroup/system.slice/docker-<container-id>.scope/memory.max

# For v1, check memory limit
cat /sys/fs/cgroup/memory/docker/<container-id>/memory.limit_in_bytes

# Check if swap accounting is enabled (v2)
cat /sys/fs/cgroup/memory.swap.current
# If file doesn't exist, swap accounting is off

Resource Limits in Docker Compose and Kubernetes

Docker Compose uses the same syntax as docker run under deploy.resources. For example: ``yaml services: app: image: myapp deploy: resources: limits: cpus: '0.5' memory: 256M reservations: cpus: '0.25' memory: 128M ` In Kubernetes, resource limits are set in pod specs. But Kubernetes uses its own cgroup management (via kubelet). Docker's limits are ignored when running under Kubernetes. Instead, set resources.limits in the pod spec. Important: Kubernetes enforces limits via cgroups, but it also uses QoS classes (Guaranteed, Burstable, BestEffort) based on whether limits equal requests. For guaranteed QoS, set limits == requests`. Production gotcha: If you set CPU limits in Kubernetes, your container may be throttled even if the node has spare CPU. This is due to CFS quotas. Consider using CPU manager policies for latency-sensitive apps.

// io.thecodeforge — DevOps tutorial

version: '3.8'
services:
  web:
    image: nginx:alpine
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 128M
        reservations:
          cpus: '0.25'
          memory: 64M
    # Note: deploy.resources only works in swarm mode, not docker-compose up
    # For docker-compose up, use:
    # mem_limit: 128m
    # cpus: 0.5

When Not to Use Resource Limits

Resource limits aren't always the answer. For single-container hosts or dedicated instances, limits add overhead and complexity. If your container is the only workload, skip limits and use host-level monitoring instead. Also, for batch jobs that need maximum throughput, limits can cause unnecessary throttling. Use --cpu-shares instead of --cpus to allow bursting. For memory, if your app has predictable usage, set a reservation but no hard limit — let the kernel OOM killer handle extreme cases. But this is risky. My rule: always set memory limits (hard or soft) for any container in a multi-tenant host. CPU limits are optional for low-priority workloads.

cgroups v2 Systemd Driver Setup

When Docker runs on a cgroups v2 host with systemd as the init system, the cgroup management driver must match. Docker and containerd support two cgroup drivers: cgroupfs (direct cgroup filesystem manipulation) and systemd (cgroup management through systemd's API). On cgroups v2, the systemd driver is mandatory for proper resource accounting and delegation.

Delegate=yes requirement: systemd must Delegate=yes for the Docker service. This tells systemd to hand over cgroup subtree management to the service. Without delegation, Docker cannot create cgroup subdirectories for containers, and resource limits silently fail. Check with systemctl show docker.service --property=Delegate. If not yes, add it in the override.conf.

Driver mismatch symptoms: If Docker uses cgroupfs driver while cgroups v2 is enabled, you'll see errors like "Error registering network: failed to add interface" or resource limits that have no effect. The container runs but docker stats shows zero limits, and docker update --memory returns an error. Solution: configure "exec-opts": ["native.cgroupdriver=systemd"] in Docker daemon.json.

cgroup.controllers and cgroup.subtree_control verification: On cgroups v2, the /sys/fs/cgroup/cgroup.controllers file lists available controllers (cpu, memory, io, pids, etc.). The /sys/fs/cgroup/cgroup.subtree_control file lists controllers delegated to child cgroups. If memory is not in subtree_control, memory limits will be silently ignored — the container writes to memory.max but the kernel does not enforce it. Fix: echo '+memory' > /sys/fs/cgroup/cgroup.subtree_control (requires root). Docker's systemd delegate should handle this automatically.

Rootless Docker delegation: For rootless Docker on cgroups v2, delegation is more complex. The rootless containerd uses systemd driver but needs --systemd-cgroup flag for runc. The user's systemd user instance must have delegate permission. Rootless Docker typically relies on /sys/fs/cgroup/user.slice/user-.slice/user@.service/ — ensure these directories exist and have subtree_control configured. Without proper delegation, rootless containers cannot set resource limits.

Verification commands: Check cgroup driver with docker info | grep -i cgroup. Check controller availability with cat /sys/fs/cgroup/cgroup.controllers. Check delegation with systemctl show docker.service --property=Delegate. Test limits with docker run --memory=64m alpine stress --vm 1 --vm-bytes 65M — the container should be OOM-killed.

#!/bin/bash
# Configure cgroups v2 with systemd cgroup driver for Docker

# ── Step 1: Verify cgroup version ────────────────────────────────────────────
echo "=== cgroup Version ==="
stat -fc %T /sys/fs/cgroup/
# Expected: cgroup2fs

# ── Step 2: Check available controllers ────────────────────────────────────
echo "=== Available Controllers ==="
cat /sys/fs/cgroup/cgroup.controllers
# Expected: cpu io memory pids ...

echo "=== Subtree Control (delegated controllers) ==="
cat /sys/fs/cgroup/cgroup.subtree_control
# Expected: cpu io memory

# ── Step 3: Configure Docker to use systemd cgroup driver ────────────────────
cat <<'EOF' > /etc/docker/daemon.json
{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}
EOF

# ── Step 4: Configure systemd delegation for Docker ──────────────────────────
mkdir -p /etc/systemd/system/docker.service.d
cat <<'EOF' > /etc/systemd/system/docker.service.d/delegate.conf
[Service]
Delegate=yes
EOF

systemctl daemon-reload
systemctl restart docker

# ── Step 5: Verify configuration ─────────────────────────────────────────────
echo "=== Docker Cgroup Driver ==="
docker info | grep -i 'cgroup'
# Expected: Cgroup Driver: systemd
#           Cgroup Version: 2

echo "=== Systemd Delegate ==="
systemctl show docker.service --property=Delegate
# Expected: Delegate=yes

# ── Step 6: Test resource limits work ───────────────────────────────────────~
echo "=== Testing Memory Limit ==="
docker run --rm --memory=64m alpine:3.19 sh -c \
  "dd if=/dev/zero of=/dev/null bs=1M count=100 &>/dev/null; echo 'Exit: \$?'"
# Expected: container is OOM-killed (exits with 137)

cgroups v2 Production Migration Checklist

Migrating from cgroups v1 to cgroups v2 across a fleet of production hosts requires a phased, canary-based approach. The migration changes kernel-level resource management behavior — errors that are invisible in staging can cascade into production outages. Follow this 6-step process.

Step 1: Fleet standardization Ensure all hosts run the same Linux distribution with kernel 5.2+ (for full cgroups v2 support). Ubuntu 22.04+, RHEL 9+, Debian 12+, and Amazon Linux 2023 all default to cgroups v2. Standardize Docker version to 24.0+ and containerd to 1.7+. Audit all hosts: stat -fc %T /sys/fs/cgroup/. Any host stuck on v1 (tmpfs) blocks the migration.

Step 2: Canary rolling migration Start with 5% of non-critical hosts. Enable cgroups v2 via kernel boot parameter: add systemd.unified_cgroup_hierarchy=1 to GRUB_CMDLINE_LINUX. Reboot. After 24 hours of monitoring with zero regressions, expand to 25%, then 50%, then 100%. Maintain ability to roll back the kernel parameter for 72 hours post-migration.

Step 3: Compliance checks For each canary host, validate 8 items: (1) docker info | grep Cgroup shows Version 2 and Driver systemd, (2) systemctl show docker.service --property=Delegate is yes, (3) /sys/fs/cgroup/cgroup.controllers includes cpu, memory, io, pids, (4) classic --blkio-weight works (v2 routes it to io.weight), (5) docker run --memory=64m --cpus=0.5 actually restricts resources (test with stress), (6) monitoring agents (Prometheus node_exporter, cAdvisor, Datadog agent) report correct cgroup v2 metrics, (7) swap accounting works or is intentionally disabled, (8) no kernel warnings in dmesg related to cgroup.

Step 4: Error catalog (8+ specific errors and fix paths) 1. "Error: cgroup2: unknown option" — Container runtime too old (upgrade runc to 1.1+) 2. "permission denied writing to /sys/fs/cgroup" — Missing Delegate=yes (add to docker.service override) 3. CPU quota silently ignored — subtree_control missing cpu controller (echo '+cpu' to cgroup.subtree_control) 4. IO throttling not working — blkio controller replaced by io controller (use --device-write-bps or upgrade Docker to auto-translate) 5. "failed to create cgroup: permission denied" — rootless Docker without user delegation (configure user slice subtree_control) 6. "docker stats" shows 0% CPU for all containers — cAdvisor or node_exporter too old for v2 (upgrade to v2-compatible versions) 7. "cannot set memory limit: device or resource busy" — MemoryLimit already set by systemd slice and container tries to exceed it (remove conflicting systemd limit or reduce container limit) 8. Swap accounting disabled — /sys/fs/cgroup/memory.swap.current file missing (add swapaccount=1 to kernel cmdline)

Step 5: Monitoring and alerting post-migration Add Grafana dashboard for cgroup v2 metrics: cgroup version per node, controller availability, subtree_control state, container throttling rates. Alert on any node where cgroup version is not 2 after the migration window. Add Prometheus recording rules for cgroup_v2_errors_total.

Step 6: Documentation and training Update runbooks to reference v2 paths (memory.max, cpu.max, io.max) instead of v1 paths. Train incident responders on the error catalog. Keep v1 fallback hosts for 1 month post-complete migration. Mark v1-compatibility tickets as resolved only after all monitoring agents confirm v2-only operation for 7 consecutive days.

#!/bin/bash
# cgroups v2 production migration — canary rollout and verification

# ── Step 1: Check current state ──────────────────────────────────────────────
echo "=== Pre-Migration Audit ==="
for host in $(cat fleet.txt); do
  CGROUP_V=$(ssh "$host" "stat -fc %T /sys/fs/cgroup/")
  DOCKER_V=$(ssh "$host" "docker version --format '{{.Server.Version}}'")
  echo "$host: cgroup=$CGROUP_V docker=$DOCKER_V"
done

# ── Step 2: Enable cgroups v2 (for canary hosts) ────────────────────────────
# Edit /etc/default/grub:
# GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX systemd.unified_cgroup_hierarchy=1"
grub2-mkconfig -o /boot/grub2/grub.cfg
reboot

# ── Step 3: Post-reboot validation ──────────────────────────────────────────
echo "=== Post-Migration Validation ==="

echo "1. cgroup version:"
stat -fc %T /sys/fs/cgroup/

echo "2. Docker cgroup info:"
docker info --format '{{.CgroupDriver}} {{.CgroupVersion}}'

echo "3. Systemd delegation:"
systemctl show docker.service --property=Delegate

echo "4. Available controllers:"
cat /sys/fs/cgroup/cgroup.controllers

echo "5. Subtree control:"
cat /sys/fs/cgroup/cgroup.subtree_control

echo "6. Test memory limit:"
docker run --rm --memory=32m alpine:3.19 sh -c \
  "dd if=/dev/zero of=/dev/null bs=1M count=50 &>/dev/null; echo exit=\$?"

echo "7. Test CPU limit:"
docker run --rm --cpus=0.5 alpine:3.19 sh -c \
  "timeout 5 sh -c 'while true; do :; done'" &
PID=$!
sleep 2
CPU=$(docker stats --no-stream --format '{{.CPUPerc}}' "$PID")
echo "CPU at --cpus=0.5: $CPU (should be ~50%)"
wait $PID 2>/dev/null

echo "8. dmesg cgroup warnings:"
dmesg | grep -i cgroup | grep -i error | head -10

Systemd Slice Resource Pools

Systemd slices provide hierarchical resource partitioning at the host level — long before Docker cgroups apply. A custom systemd slice acts as a resource pool that groups containers by team, environment, or criticality, enabling dual-layer resource management: slice limits cap the pool, Docker cgroups divide the pool.

Custom slice units: Create a slice file /etc/systemd/system/production.slice that defines a resource pool for all production containers. The slice enforces a total memory budget, CPU share, and I/O weight for all containers within it. When a new container starts under this slice, systemd automatically places it in the slice's cgroup subtree.

MemoryWeight and IOWeight: These parameters control proportional distribution within a slice. MemoryWeight=100 (range 1-10000) determines memory reclaim priority — a process with weight 200 is twice as likely to be reclaimed under pressure as one with weight 100. IOWeight=1000 (range 1-10000) sets the proportional I/O bandwidth. Unlike container-level --blkio-weight, slice-level IOWeight applies to all I/O (including buffered writes) because systemd uses io.weight on cgroups v2.

MemoryHigh vs MemoryMax: MemoryHigh is a soft limit — when exceeded, the kernel aggressively reclaims memory from the slice's processes but does not kill them. MemoryMax is a hard limit — processes are OOM-killed within the slice when exceeded. Use MemoryHigh for non-critical pools (background jobs, batch processing) where throttling is preferred over killing. Use MemoryMax for critical pools (production APIs) where resource overcommit must never happen.

Dual-layer management example: - Layer 1 (slice): production.slice with MemoryMax=16G, CPUQuota=800%, IOWeight=1000 - Layer 2 (Docker): containers within the slice with --memory=512m, --cpus=0.5

The slice guarantees that all production containers combined never exceed 16GB. Docker cgroups then divides the 16GB among individual containers. If a container is misconfigured with --memory=4g, it takes 4GB of the 16GB pool, but cannot exceed the pool.

Use case: multi-tenant isolation: Create slices per team: team-api.slice, team-data.slice, team-batch.slice. Each team gets a fixed resource pool. A noisy neighbor in team-batch cannot starve team-api because the slices are isolated at the kernel level. This is more reliable than relying on container-level limits alone, which operate within shared cgroup hierarchies.

Configuration: Create /etc/systemd/system/.slice with [Slice] section. Assign containers using --cgroup-parent pointing to the slice in cgroups v2 syntax: docker run --cgroup-parent=/production.slice container:tag. Verify with systemd-cgls or cat /proc/self/cgroup.

#!/bin/bash
# Systemd slice resource pool setup for Docker containers

# ── Create a production slice with 16GB max memory, 8 CPU cores, high IO ────
cat <<'EOF' > /etc/systemd/system/production.slice
[Unit]
Description=Production Container Resource Pool
Before=docker.service

[Slice]
# Soft limit: above 12GB, aggressive reclaim
MemoryHigh=12G
# Hard limit: at 16GB, OOM kills within slice
MemoryMax=16G
# CPU: limit to 8 cores (800%)
CPUQuota=800%
# Memory reclaim weight (higher = less reclaim, default 100)
MemoryWeight=200
# IO weight (higher = more bandwidth)
IOWeight=1000
EOF

# ── Create a batch slice with relaxed limits ─────────────────────────────────
cat <<'EOF' > /etc/systemd/system/batch.slice
[Unit]
Description=Batch Job Container Resource Pool

[Slice]
MemoryHigh=4G
MemoryMax=8G
CPUQuota=400%
MemoryWeight=50
IOWeight=200
EOF

# ── Reload systemd and activate slices ───────────────────────────────────────
systemctl daemon-reload
systemctl start production.slice batch.slice

# ── Run containers within slices ────────────────────────────────────────────
# Production API (goes into production.slice)
docker run -d --name prod-api \
  --cgroup-parent=/production.slice \
  --memory=512m \
  --cpus=0.5 \
  nginx:alpine

# Batch processor (goes into batch.slice)
docker run -d --name batch-processor \
  --cgroup-parent=/batch.slice \
  --memory=1g \
  --cpus=1.0 \
  alpine:3.19 sleep 3600

# ── Verify slice assignment ──────────────────────────────────────────────────
echo "=== Cgroup Tree ==="
systemd-cgls
# Expected output:
# /production.slice/
#   docker-<prod-api-id>.scope/
# /batch.slice/
#   docker-<batch-id>.scope/

echo "=== Slice Resource Limits ==="
systemctl show production.slice --property=MemoryMax --property=CPUQuota --property=IOWeight
systemctl show batch.slice --property=MemoryMax --property=CPUQuota --property=IOWeight

echo "=== Container Cgroup Paths ==="
docker inspect prod-api --format '{{.HostConfig.CgroupParent}}'
docker inspect batch-processor --format '{{.HostConfig.CgroupParent}}'

cgroups v2 Error Troubleshooting

cgroups v2 introduces new error modes that differ from v1. These errors manifest in Docker command failures, silent limit violations, and resource accounting discrepancies. This section covers the 5 most common v2-specific errors with exact diagnosis and fix paths.

Error 1: "cgroup2: unknown option" - Symptom: Docker run fails with "unknown option" referencing cgroup2. - Cause: Container runtime (runc) version < 1.1 does not support cgroups v2. Docker Engine 20.10+ bundles runc 1.1+, but older versions or custom runc installations may lag. - Diagnosis: runc --version | grep runc — must be >= 1.1.0. Check rootless containerd's runc path. - Fix: Upgrade runc to 1.1+. On Ubuntu: apt install containerd (bundles runc). On RHEL: dnf update container-selinux. Verify with docker run --rm alpine:3.19 echo ok.

Error 2: "permission denied writing to /sys/fs/cgroup" - Symptom: Container creation fails with permission denied on cgroup files. - Cause: systemd Delegate=yes not set on docker.service. Docker's cgroup driver tries to write to /sys/fs/cgroup/system.slice/docker-<container>.scope/ but systemd owns that subtree. - Diagnosis: systemctl show docker.service --property=Delegate shows 'no'. - Fix: Add Delegate=yes to docker.service override, systemctl daemon-reload && systemctl restart docker.

Error 3: CPU quota silently ignored - Symptom: Container runs with --cpus=0.5 but uses 200% CPU. docker stats shows >100% CPU. - Cause: cpu controller not enabled in cgroup.subtree_control. The kernel accepts writes to cpu.max but does not enforce them. - Diagnosis: cat /sys/fs/cgroup/cgroup.subtree_control does not include 'cpu'. - Fix: echo '+cpu' > /sys/fs/cgroup/cgroup.subtree_control. If this fails, add CPUAccounting=true to docker.service's systemd override. For persistent fix, ensure Delegate=yes is set.

Error 4: IO throttling not working (blkio vs cgroups v2 IO controller) - Symptom: --device-write-bps or --blkio-weight has no effect. Container I/O saturates the disk. - Cause: cgroups v2 replaces the blkio controller with the io controller. Docker 24.0+ auto-translates --device-write-bps to io.max writes. Older Docker versions write to blkio controller files that have no effect on v2. - Diagnosis: docker info | grep Cgroup shows Version 2. Write test: docker run --rm --device-write-bps /dev/sda:1mb alpine:3.19 sh -c "dd if=/dev/zero of=/tmp/test bs=1M count=100 oflag=direct". If speed > 1MB/s, throttling is not working. - Fix: Upgrade Docker to 24.0+. On v2, use cgroups v2 native flags: --io-max-write-bps /dev/sda:1mb (experimental in newer Docker). Alternative: configure IO limits at the systemd slice level via IOWeight.

Error 5: Memory limit not enforced for rootless containers - Symptom: docker run --memory=128m runs fine even with 256MB+ usage. No OOM. - Cause: Rootless Docker on cgroups v2 requires user-specific delegation. The user's systemd user slice must have subtree_control configured. - Diagnosis: As rootless user, check cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers. If limited, delegation is incomplete. - Fix: As root, run echo '+memory +cpu +io' > /sys/fs/cgroup/user.slice/user-$(id -u).slice/cgroup.subtree_control. For persistent config, add to /etc/systemd/system/user@.service.d/delegate.conf.

General v2 debugging workflow: When any resource limit seems broken, run this sequence: 1. stat -fc %T /sys/fs/cgroup/ — confirm you're on v2 2. docker info | grep -i cgroup — confirm driver is systemd 3. cat /sys/fs/cgroup/cgroup.controllers — what's available 4. cat /sys/fs/cgroup/cgroup.subtree_control — what's delegated 5. systemctl show docker.service --property=Delegate — delegation status 6. Apply a known limit (docker run --memory=32m alpine stress --vm 1 --vm-bytes 33M) and check if it OOMs 7. If not OOM'd, the controllers are not being enforced — fix subtree_control

#!/bin/bash
# cgroups v2 error troubleshooting toolkit

# ── Error 1: 'cgroup2: unknown option' ───────────────────────────────────────
echo "=== Error 1: Unknown Option ==="
runc --version 2>/dev/null || echo "runc not found"
docker version --format '{{.Server.Version}}'
# Fix: upgrade runc >= 1.1, Docker >= 20.10

# ── Error 2: Permission Denied ───────────────────────────────────────────────
echo "=== Error 2: Permission Denied ==="
systemctl show docker.service --property=Delegate
# If not 'yes', fix:
# mkdir -p /etc/systemd/system/docker.service.d
# echo -e '[Service]\nDelegate=yes' > /etc/systemd/system/docker.service.d/delegate.conf
# systemctl daemon-reload && systemctl restart docker

# ── Error 3: CPU Quota Silently Ignored ─────────────────────────────────────
echo "=== Error 3: CPU Quota Check ==="
echo "subtree_control: $(cat /sys/fs/cgroup/cgroup.subtree_control)"
echo "Testing CPU limit..."
CID=$(docker run -d --cpus=0.5 alpine:3.19 sh -c 'while true; do :; done')
sleep 3
CPU=$(docker stats --no-stream --format '{{.CPUPerc}}' "$CID")
echo "CPU usage with --cpus=0.5: $CPU"
docker kill "$CID" >/dev/null 2>&1
# If CPU > 70%, subtree_control is missing 'cpu'

# ── Error 4: IO Throttling Not Working ───────────────────────────────────────
echo "=== Error 4: IO Throttling Check ==="
echo "Docker version: $(docker version --format '{{.Server.Version}}')"
echo "Cgroup version: $(docker info --format '{{.CgroupVersion}}')"
if command -v dd &>/dev/null; then
  echo "Testing IO throttling..."
  docker run --rm --device-write-bps /dev/sda:1mb \
    alpine:3.19 sh -c "dd if=/dev/zero of=/tmp/test bs=1M count=50 oflag=direct 2>&1 | grep -o '[0-9.]\+ MB/s'"
fi

# ── Error 5: Memory Limit Not Enforced (rootless) ───────────────────────────
echo "=== Error 5: Memory Limit Check ==="
echo "User: $(whoami)"
if [ "$(id -u)" -ne 0 ]; then
  USER_SLICE="/sys/fs/cgroup/user.slice/user-$(id -u).slice"
  if [ -d "$USER_SLICE" ]; then
    echo "User slice controllers: $(cat ${USER_SLICE}/cgroup.controllers 2>/dev/null || echo 'N/A')"
  fi
fi

# ── Universal test: Does any memory limit work? ─────────────────────────────
echo "=== Universal Limit Test ==="
docker run --rm --memory=32m alpine:3.19 sh -c \
  "dd if=/dev/zero of=/dev/null bs=1M count=50 &>/dev/null; echo 'Exit code: \$?'"
# Exit code 137 = OOM, limits work. Exit code 0 = limit not enforced.