Docker No Space Left on Device: Reclaim Disk

It starts as a pull that dies halfway: "failed to register layer: no space left on device." Then builds fail, then running containers can't write, then the daemon itself starts acting strange. Your monitoring said disk was at 62% last week — but that was the root filesystem, and Docker lives on its own volume (or worse, shares /var with logs that exploded overnight). Docker disk exhaustion is a slow leak that becomes a sudden outage, and it always lands during a deploy.

The frustration is that docker system df and df -h disagree, prune commands free less than expected, and the same failure returns a month later. That's because Docker stores five different kinds of bytes — images, containers, volumes, build cache, and logs — and each needs its own cleanup. Pruning images when the problem is 40GB of json-file logs fixes nothing.

This guide maps every byte to its owner: diagnose with df plus docker system df, reclaim with targeted prunes, cap logs with max-size, and move data-root when the volume is genuinely undersized. You'll leave with a cleanup runbook and the guards that keep the floor swept.

Where Docker's Bytes Actually Live

Docker's disk footprint splits into five buckets, and each fills for its own reason. overlay2 stores image layers and container writable layers — it grows with every pulled tag and every file your containers write. volumes holds named volume data, which survives container deletion by design and is the classic "I deleted everything, disk still full" surprise. containers holds per-container state plus json-file logs, which grow without bound unless capped. buildkit holds build cache — intermediate layers from every build, invisible in docker images, often the biggest bucket on CI. And tmp plus network sandboxes round out the remainder.

The data-root (/var/lib/docker unless moved) sits on one filesystem, and that filesystem is shared with whatever else lives on the volume — system logs, other apps, or nothing if you're lucky. df -h reports the filesystem; docker system df reports Docker's accounted portion. The gap between them is the first clue: a big gap means logs, another tenant, or held-open deleted files. Small gap means Docker's own buckets, and docker system df -v names the winner.

Learn the weighing commands before the deleting commands. sudo du -sh /var/lib/docker/ ranks the buckets on disk truthfully, docker builder du isolates build cache, and the find over -json.log ranks log files. Weighing takes a minute and aims every later command. Teams that prune blind delete 11GB of images against an 87GB cache problem; teams that weigh fix it in one shot.

Diagnose: df, system df, and du in 60 Seconds

Run the triage trio in order and you'll know the bucket within a minute. First df -h /var/lib/docker (or df -h / on shared-root boxes): Use% near 100 confirms ENOSPC territory and names the filesystem. Also glance at df -i — if inodes are exhausted while blocks remain, millions of tiny files (usually runaway logs or caches) are the story, and pruning whole objects beats deleting files. Second, docker system df: TYPE, TOTAL, ACTIVE, SIZE, RECLAIMABLE per bucket, with build cache listed separately on modern versions. RECLAIMABLE is your estimated prize for each prune.

Third, docker system df -v drills into the winners — which images are dangling, which containers stopped weeks ago, which volumes unreferenced. Pair it with sudo du -sh /var/lib/docker/ | sort -rh to catch what Docker under-reports, especially logs. When system df claims 20GB but df shows 95GB used, the du ranking plus the -json.log hunt finds the dark matter: logs, held-open deleted files (lsof +L1), or a non-Docker tenant on the same volume.

Write the three outputs into the incident thread before acting. They take 60 seconds, they make the fix reviewable, and they stop the second-guessing about whether the prune was safe — RECLAIMABLE plus the stopped-container list is your evidence. On Kubernetes nodes the same trio runs against containerd's paths, but the logic is identical: filesystem first, runtime accounting second, directory weights third.

# 1. Filesystem truth (blocks + inodes)
df -h /var/lib/docker
df -i /var/lib/docker

# 2. Docker's own accounting per bucket
docker system df
docker system df -v 2>/dev/null | head -40

# 3. Directory weights catch what accounting hides
sudo du -sh /var/lib/docker/* 2>/dev/null | sort -rh | head
sudo find /var/lib/docker/containers -name '*-json.log' -exec du -h {} + 2>/dev/null | sort -rh | head -10

# Deleted-but-held-open files pinning freed space
sudo lsof +L1 2>/dev/null | grep -iE 'docker|containerd' | head

Prune Safely: Containers, Images, Volumes, Networks

Prune in ascending order of danger. Stopped containers first: docker container prune reclaims their writable layers and is nearly always safe — stopped means nobody needs them, though check docker ps -a for containers you stopped deliberately as checkpoints. Then dangling images: docker image prune drops untagged layers from rebuilds; adding -a also drops tagged images no container uses, which is safe on builders but think twice on deploy hosts mid-release. Volumes last and carefully: docker volume prune deletes every volume no container references, including database data from a compose stack that's merely down rather than gone.

The nuclear option, docker system prune -a --volumes, combines all three plus networks and build cache — powerful and unforgiving. Gate it with filters: --filter 'until=168h' keeps the last week, and label filters protect blessed objects (docker volume ls --filter label=keep). On shared hosts, list candidates first (docker images, docker volume ls) and prune by name rather than by flag. A filter typo with -f is how teams delete the production database volume at 2 AM.

After pruning, re-run docker system df to confirm the reclaim matched the estimate, and retry the failed pull or build. If usage barely moved, you pruned the wrong bucket — go back to du instead of pruning harder. And remember pruning is triage: it buys time while you add the caps and schedules below, not a strategy you repeat monthly by hand.

# Ascending danger: containers, dangling images, then aged unused images
docker ps -a --format '{{.Names}} {{.Status}}' | head -20
docker container prune -f
docker image prune -f
docker image prune -af --filter 'until=720h'   # unused tags older than 30d

# Volumes: list before you leap (down stacks still own data)
docker volume ls
docker volume prune --filter 'label!=keep'   # drops unreferenced volumes

# Networks rarely matter, but they're free to sweep
docker network prune -f

# Confirm the reclaim, then retry the failed operation
docker system df
docker pull myapp:latest

Build Cache: the Bucket Everyone Forgets

BuildKit caches every intermediate layer from every build so rebuilds are fast — and never evicts anything unless configured. On a CI builder running dozens of builds daily, cache grows monotonically for months: 14 months reached 87GB in the Tuesday incident. docker images doesn't show it, docker image prune -a doesn't touch it, and older docker system df versions bury it. The only commands that see it are docker builder du (or docker buildx du) and the build-cache line in modern system df output. If your builders have never run a builder prune, assume tens of gigabytes are sitting there.

Reclaim with docker builder prune, surgically or fully: --keep-storage 20GB retains recent cache for speed while dropping the tail, while -af wipes everything and makes the next build slow but correct. A full wipe during business hours on a busy builder farm can spike build times 3-5x for an hour — schedule it or use keep-storage. For permanence, set a GC ceiling in daemon.json so the daemon evicts automatically: the builder GC policy with keepStorage caps total cache without any cron.

Also shrink what enters the cache. A .dockerignore that excludes .git, node_modules, and test fixtures cuts context and cache churn; multi-stage builds with pinned builder images avoid re-pulling shifting bases. Cache is a performance feature with a storage bill — budget it like one, with a cap in config and a prune in the weekly schedule, and it stops being an incident source.

# See the invisible bucket
 docker builder du
# docker buildx du   # equivalent on newer CLIs

# Surgical: keep 20GB of hot cache, drop the tail
docker builder prune --keep-storage 20GB -f

# Full wipe (next builds go slow for a while — schedule wisely)
# docker builder prune -af

# Permanent ceiling in /etc/docker/daemon.json, then restart
# {
#   "builder": { "gc": { "defaultKeepStorage": "20GB", "enabled": true } }
# }
sudo systemctl restart docker
docker builder du   # confirm the ceiling holds

Log Caps: Stop json-file From Growing Forever

The default json-file log driver appends every container stdout line to a -json.log file with no rotation unless you configure it. A chatty service at a few MB per hour reaches gigabytes in weeks; a debug-logging incident can do it in a day. These logs live under /var/lib/docker/containers, count opaquely in system df, and survive container restarts — only container removal (or rotation) reclaims them. When the -json.log hunt shows multi-GB files, you've found the leak that pruning images will never fix.

Cap all future containers in daemon.json: log-driver json-file with log-opts max-size (per file) and max-file (rotated files kept). Fifty megabytes times 3 files bounds any container at 150MB — tune to your log volume and retention needs. Restart dockerd after the change, and understand the boundary: caps apply to containers created after the change. Existing containers keep their uncapped files until recreated, so a compose up --force-recreate (or pod rollout) is part of the rollout.

For immediate relief without recreation, truncate the live file: truncate -s caps the size while the container keeps its file handle, unlike rm which leaves the space pinned until restart. Longer term, ship logs to a real backend (local driver with rotation, syslog, or a collector) instead of hoarding them on the node. Node disks are the most expensive log store you own — bounded local files plus centralized aggregation is the production pattern.

# Rank the log files eating the disk
sudo find /var/lib/docker/containers -name '*-json.log' -exec du -h {} + 2>/dev/null | sort -rh | head -10

# Cap future containers: 50MB x 3 files each (daemon.json), then restart
sudo python3 - <<'EOF'
import json
p = '/etc/docker/daemon.json'
try:
    cfg = json.load(open(p))
except FileNotFoundError:
    cfg = {}
cfg['log-driver'] = 'json-file'
cfg['log-opts'] = {'max-size': '50m', 'max-file': '3'}
json.dump(cfg, open(p, 'w'), indent=2)
print(open(p).read())
EOF
sudo systemctl restart docker

# Immediate relief on a live container (keeps the file handle valid)
CID=$(docker ps -q --filter name=myapp | head -1)
sudo truncate -s 100M /var/lib/docker/containers/${CID}/${CID}-json.log

# Recreate so existing containers pick up the caps
docker compose up -d --force-recreate app

Move data-root and Never Page for This Again

Sometimes the volume is simply too small: Docker shares a 20GB root disk with the OS, or the team outgrew the original sizing. Pruning then becomes a weekly chore that always lands at the worst time. Moving the data-root to a dedicated, sized volume ends the treadmill. Provision a volume (100GB minimum for builders, more for heavy CI), format and mount it at /mnt/docker-data, set data-root in daemon.json, and migrate: stop dockerd, rsync -a /var/lib/docker/ to the new mount, start dockerd, and verify with docker info --format '{{.DockerRootDir}}' plus a hello-world run. Keep the old directory until the next deploy proves the move.

Then automate the sweeping. A weekly cron running image and builder prunes with until-filters, a builder GC ceiling, log caps, and a disk alert at 75% with the runbook attached — together these four convert disk-full from an incident into background hygiene. Alert on rate-of-fill too when you can: 75% reached in a day means a leak worth investigating, while 75% reached over months is routine growth.

Verify the whole stack after any change: docker info names the new root dir, system df shows headroom, a fresh pull and build succeed, and the cron logs its runs somewhere you'll notice failures. Disk is infrastructure with an owner, a size, a growth plan, and a monitor. Give Docker's bytes all four and this error retires to a story you tell new hires.

# Provision + mount the new volume first (example: /dev/nvme1n1)
sudo mkfs -t ext4 /dev/nvme1n1
sudo mkdir -p /mnt/docker-data
sudo mount /dev/nvme1n1 /mnt/docker-data
df -h /mnt/docker-data

# Point the daemon at it
sudo python3 - <<'EOF'
import json
p = '/etc/docker/daemon.json'
try:
    cfg = json.load(open(p))
except FileNotFoundError:
    cfg = {}
cfg['data-root'] = '/mnt/docker-data'
json.dump(cfg, open(p, 'w'), indent=2)
print(open(p).read())
EOF

# Migrate: stop, copy, start, verify (keep the old dir one deploy)
sudo systemctl stop docker
sudo rsync -a /var/lib/docker/ /mnt/docker-data/
sudo systemctl start docker
docker info --format 'root={{.DockerRootDir}}'
docker run --rm hello-world | head -3

# Weekly hygiene cron
# 0 3 * * 0 /usr/bin/docker image prune -af --filter 'until=168h' && /usr/bin/docker builder prune -f --keep-storage 20GB