OCI Runtime Create Failed: Fix runc Errors

You run docker run, and instead of logs you get a wall of text ending in "OCI runtime create failed: runc create failed: unable to start container process." Your app never printed a line. Your code never executed. The failure happened in the few hundred milliseconds between the daemon accepting your request and your process starting — inside the runtime layer most engineers never look at. The instinct is to debug the application. That's the wrong layer entirely.

This error is Docker telling you the container couldn't be assembled: a mount source that doesn't exist, an entrypoint binary that isn't there, a cgroup setting the kernel rejected, or an image layer that fails its checksum. The message is verbose because three layers (daemon, containerd shim, runc) each add their complaint, but the actual cause is almost always in the last two lines.

This guide teaches you to read the error tail first, then work the four usual suspects in order: bind paths, entrypoint, cgroup driver, and corrupt layers. You'll get the inspect commands that reveal each one and the fixes that hold up in production.

Read the Tail: Three Layers, One Real Complaint

An OCI create error reads like three errors stacked in a trench coat. dockerd wraps containerd's message, containerd wraps the shim's, and the shim wraps runc's — so the first line you see ("docker: Error response from daemon") carries zero information. Scroll to the last two lines. That's runc speaking, and runc is specific: it names the syscall, the path, and the errno. "mounting /data/uploads to rootfs at /app/uploads caused: stat /data/uploads: no such file or directory" is a complete diagnosis in one sentence, if you read it instead of the wrapping.

Build the habit of capturing the full text: docker run ... 2>&1 | tail -5 preserves the tail while the terminal scrollback eats it. Then classify the tail into one of four buckets — mount, exec, cgroup, or layer — because each bucket has its own section below with its own commands. Guessing across buckets is how a 5-minute mount fix becomes a 2-hour image rebuild saga.

The second habit: reproduce minimally. Strip your 200-line compose service down to docker run with just the suspect flag. If docker run -v /data/uploads:/app/uploads:ro myapp:latest true fails the same way, you've isolated the mount from every other variable. Minimal reproduction is the difference between debugging one thing and debugging twelve things that share an error message.

Bad Bind Mounts: the Missing Host Path

Bind mounts graft a host path into the container, and runc stats the source before mounting. If the directory doesn't exist, the create fails — modern Docker refuses to auto-create host paths for explicit --mount binds, and even -v behavior varies by driver. The path might be a typo (/data/upload vs /data/uploads), a volume that was never mounted on this host, an NFS share that didn't come back after reboot, or a directory the migration script skipped. The error names the exact source path, so there's no detective work — just go look at it.

Check with ls -ld on the literal path from the error, on the host where it failed. Partial-fleet failures are the signature here: the path exists on 9 nodes and not on 3, because storage, migrations, and manual fixes never apply evenly. Also check permissions: runc runs the mount as root, but a :ro bind onto a directory the container user can't read fails later at access time, and SELinux hosts need the :z or :Z relabel flag or the mount is denied outright.

Prefer named volumes over host binds for anything the app writes — volumes are daemon-managed and exist on every host identically. When you must bind (configs, device nodes, legacy data), declare the path in config management and add a pre-deploy stat check on every target node. A one-line test -d in the deploy script would have caught the Friday incident before a single container failed.

# Reproduce with the suspect flag only
docker run --rm -v /data/uploads:/app/uploads:ro myapp:latest true 2>&1 | tail -3

# Inspect the host source exactly as runc sees it
ls -ld /data/uploads
stat -c 'owner=%U perms=%a' /data/uploads
mount | grep -E 'uploads|nfs ' || echo 'no matching mount'

# SELinux host denying the mount? Check the audit trail
sestatus 2>/dev/null | head -2
sudo ausearch -m avc -ts recent 2>/dev/null | tail -5

# Fix: create with the UID your container user expects
docker inspect --format '{{.Config.User}}' myapp:latest
sudo mkdir -p /data/uploads && sudo chown 1000:1000 /data/uploads

# Retry with shared-content relabel on SELinux hosts
docker run --rm -v /data/uploads:/app/uploads:ro,Z myapp:latest true

Invalid Entrypoint: the Binary That Isn't There

After mounts succeed, runc execs your entrypoint as PID 1. If that path doesn't exist inside the image, isn't executable, or names an interpreter that's missing, creation fails with an exec error — "no such file or directory" even when the file visibly exists, which confuses everyone once. The classic trap: the script exists but its shebang names /bin/bash in an Alpine image that only ships /bin/sh, or the file lost its execute bit in a COPY from Windows. The kernel reports the interpreter as missing, and the message points at your script.

Diagnose from outside the container. docker inspect --format prints the image's declared Entrypoint and Cmd without running anything — compare that against your run flags, because CLI args replace Cmd and --entrypoint replaces Entrypoint, and the merged result is what runc actually execs. Then test with docker run --rm --entrypoint sh to get a shell: ls -l the entrypoint path, read its shebang line, and try executing it by hand. If the shell override itself fails, stop — the image is corrupt, not misconfigured.

Fix at the source. Use exec-form ENTRYPOINT with absolute paths, pin the interpreter your base image actually ships, and keep ENTRYPOINT scripts to a POSIX sh subset unless you've verified bash exists. In CI, add a smoke step that runs the image with --entrypoint sh -c 'test -x /app/start.sh' so a broken entrypoint fails the build, not the deploy.

# What will runc actually exec? (image defaults + your flags merged)
docker inspect --format 'entrypoint={{json .Config.Entrypoint}} cmd={{json .Config.Cmd}}' myapp:latest

# Get a shell in and interrogate the entrypoint by hand
docker run --rm --entrypoint sh myapp:latest -c 'ls -l /app/start.sh; head -1 /app/start.sh'

# Missing interpreter? Compare shebang vs what the image ships
docker run --rm --entrypoint sh myapp:latest -c 'ls -l /bin/sh /bin/bash 2>&1'

# Lost exec bit (common after COPY from Windows checkouts)?
docker run --rm --entrypoint sh myapp:latest -c 'test -x /app/start.sh && echo EXEC-OK || echo MISSING-X-BIT'

# Fix in the Dockerfile, not at runtime:
# ENTRYPOINT ["/app/start.sh"] with '#!/bin/sh' first line + chmod +x

Cgroup Driver Mismatch: systemd vs cgroupfs

On a plain docker run host, cgroup settings rarely bite. Under Kubernetes they bite hard. The kubelet and the container runtime must use the same cgroup driver — systemd on every modern distro — or container creation fails with cgroup-flavored errors about paths, parents, or permissions under /sys/fs/cgroup. The mismatch usually arrives via an old daemon.json carrying native.cgroupdriver=cgroupfs, or a kubelet installed from a guide written for the cgroupfs era. Docker Desktop and fresh installs default correctly, which is why the bug only appears on hand-configured nodes.

Confirm both sides independently. docker info --format '{{.CgroupDriver}}' prints the daemon's driver; on the node, check the kubelet config or ps aux | grep kubelet for --cgroup-driver. If they disagree, change the daemon side — Kubernetes standardized on systemd years ago, so the daemon should follow. Set exec-opts to native.cgroupdriver=systemd in /etc/docker/daemon.json, validate the JSON, restart dockerd, and re-check docker info.

Treat driver config as fleet state, not per-node folklore. Manage daemon.json through config management, assert the driver in node conformance checks, and be suspicious of any tutorial that sets cgroupfs — it's a time traveler from 2019. After the fix, redeploy a test pod on each touched node; cgroup changes only affect containers created after the restart.

# Both sides must agree (want: systemd on each)
docker info --format 'docker cgroup driver: {{.CgroupDriver}}'
ps aux | grep '[k]ubelet' | grep -o '\-\-cgroup-driver=[a-z]*'

# Align the daemon to systemd via daemon.json
sudo python3 - <<'EOF'
import json
p = '/etc/docker/daemon.json'
try:
    cfg = json.load(open(p))
except FileNotFoundError:
    cfg = {}
cfg['exec-opts'] = ['native.cgroupdriver=systemd']
json.dump(cfg, open(p, 'w'), indent=2)
print(open(p).read())
EOF

sudo systemctl restart docker
docker info --format 'docker cgroup driver: {{.CgroupDriver}}'

Corrupted Image Layers: Pull Fresh

When mounts, entrypoint, and cgroups all check out, suspect the bytes. Layers corrupt in transit (flaky registry mirror, interrupted pull resumed badly), at rest (dying disk, full filesystem mid-extract), or at build time (daemon crash during build leaving a bad layer in the local cache). The symptoms vary — checksum mismatches, failed to register layer, unexpected EOF during create — but the test is uniform: the same tag fails identically with a minimal run while a fresh pull elsewhere works. If the image fails on one host and runs on five others, the host's cached layers are guilty, not the registry.

The fix is a clean slate on the failing machine: remove the image, pull fresh, and watch the pull output for retries or checksum errors that confirm the theory. docker rmi needs the image unused — check docker ps -a for stopped containers pinning it before reaching for -f. After pulling, compare the RootFS layer digests against a healthy host; identical digests with different behavior means the problem was never the image, so go back to host facts.

If corruption recurs on the same host, stop pulling and check the disk: df -h for full filesystems, dmesg for I/O errors, SMART data for a dying drive. Recurring single-host corruption is hardware or filesystem trouble, and no amount of re-pulling fixes hardware. Pin production to digests rather than mutable tags while you're at it — a digest guarantees every host unpacks the same bytes.

# Who pins the image? (rmi refuses while containers reference it)
docker ps -a --filter ancestor=myapp:latest --format '{{.Names}} {{.Status}}'

# Clean slate on the failing host, watching pull output for retries
docker rmi myapp:latest
docker pull myapp:latest 2>&1 | tee /tmp/pull.log
grep -iE 'retry|checksum|error' /tmp/pull.log || echo 'pull clean'

# Compare layer digests against a healthy host
docker inspect --format '{{.RootFS.Layers}}' myapp:latest

# Recurring on one host? Suspect disk, not registry
df -h /var/lib/docker
dmesg | tail -20 | grep -iE 'I/O error|Buffer|EXT4|nvme|sd[a-z]' || echo 'no disk errors'

Inspect, Logs, and the Lock-In

Two commands close every OCI investigation: docker inspect and the daemon logs. docker inspect --format renders the merged runtime config — mounts, entrypoint, user, cgroup-relevant limits — exactly as the daemon will hand it to runc. Reading it before running catches typos in -v sources, wrong entrypoint paths, and user IDs with no home. The daemon logs (journalctl -u docker, or containerd logs under Kubernetes) add the daemon's view: which API call failed and what the runtime reported back. Together they replace guessing with reading.

Verification is a ladder: the minimal failing run now succeeds, docker inspect shows the corrected config, and the full compose service or pod starts and passes health checks. On Kubernetes, follow with kubectl describe pod and kubectl logs to confirm the runtime layer is green before declaring victory. Each rung tests a wider scope, so a pass at the top means the fix is real.

Lock it in with three guards. Pre-deploy checks that stat every bind source on every target node. CI smoke steps that boot the image with its real entrypoint. And digest-pinned production tags so every host assembles the same bytes. OCI create failures are assembly failures — make the inputs (paths, entrypoints, configs, bytes) declared and verified, and the assembly stops failing.

# Read the merged runtime config before you run
docker inspect --format 'mounts={{json .Mounts}} user={{.Config.User}}' myapp-container 2>/dev/null

# Daemon's view of the last failure
sudo journalctl -u docker --since '10 min ago' | grep -i -B2 -A5 'runc.*fail\|create.*fail' | tail -25

# Verification ladder: minimal run, then the real workload
docker run --rm myapp:latest true && echo MINIMAL-OK
docker compose up -d app && sleep 5 && docker compose ps app

# Kubernetes follow-up when the runtime sits under kubelet
kubectl describe pod -l app=myapp | grep -A5 Events | tail -8
kubectl logs -l app=myapp --tail=5