Too Many Open Files - Raise Limits and Fix Leaks

Your app starts refusing connections and the logs say Too many open files. Restarts help for a day, then it returns. The cause is almost never disk space: it is the file descriptor limit. Every open file, socket, and pipe consumes one descriptor, and each process has a cap set by ulimit -n. Busy servers with connection churn hit the default 1024 fast.

The trap is raising the limit without asking why usage climbs. A leak that opens sockets and never closes them will eat 65,000 descriptors as happily as 1,024. Blindly doubling limits buys time while hiding the bug. Production-worthy fixes do both: set correct limits at every layer, then prove with lsof that descriptor counts stay flat under load.

This guide walks the full stack: what descriptors are, soft vs hard ulimit, limits.conf for login shells, systemd LimitNOFILE overrides that most teams miss, leak hunting with lsof -p and /proc/PID/fd, and why sockets count against the same budget. You will leave with persistent limits and a leak-hunt routine.

File Descriptors Explained - Files, Sockets, and Pipes

A file descriptor is the kernel ticket for anything a process has open: regular files, TCP and Unix sockets, pipes, eventpoll sets, even /dev/null. open, socket, accept, and pipe each consume one integer slot in the process table. close returns it. The per-process count lives in /proc/PID/fd, one symlink per live descriptor, and the ceiling lives in /proc/PID/limits under Max open files. When the count reaches the ceiling, the next open fails with EMFILE and your logs say Too many open files. Nothing about disk space matters here: a terabyte of free disk does not grant one more descriptor. Two ceilings stack. The per-process soft limit is what the app hits day to day; the hard limit caps how high the soft one can be raised without root. Above both sits fs.file-max, the system-wide budget shared by every process. Hitting file-max prints VFS: file-max limit reached in dmesg and breaks all processes at once. Normal servers never approach it, but containers sharing a host kernel can surprise you. Learn to read the three numbers together: process use from /proc/PID/fd, process ceiling from ulimit -n, system pressure from /proc/sys/fs/file-nr. That trio tells you in seconds whether one app is greedy or the whole box is tired.

ulimit -n
ulimit -Hn
cat /proc/sys/fs/file-nr
cat /proc/sys/fs/file-max
ls /proc/$$/fd | wc -l

ulimit Soft vs Hard - Check and Raise Without Lockout

ulimit -n shows the soft limit: the working ceiling your shell and its children obey right now. ulimit -Hn shows the hard limit: the maximum anyone without root can raise the soft one to. Raising soft up to hard needs no privilege: ulimit -n 4096 just works. Raising hard needs root, and lowering hard is one-way for mortals, so be careful in shared sessions. These values are per-process and inherited at fork: changing them in your terminal affects only that shell and what it starts, never a running daemon or a systemd service. That is why ulimit fixes that work in testing vanish in production. Verify the real target instead of your shell: cat /proc/PID/limits shows the service actual soft and hard ceilings. If your shell says 65,535 but the service says 1,024, you fixed the wrong process. For interactive work, set both sensibly in the shell profile. For daemons, note the numbers here and carry them into limits.conf or the systemd unit in the next sections. Never set unlimited in production: one runaway process can then starve the whole host of descriptors instead of failing alone and noisily. Record the chosen soft and hard values in your runbook with the reasoning behind them. The next engineer should see numbers plus justification, not mystery digits.

ulimit -Sn
ulimit -Hn
cat /proc/1234/limits | grep -i open
ulimit -n 4096
ulimit -Sn

limits.conf and PAM - Make Login Limits Persistent

Shell ulimit changes evaporate on logout, so persistent interactive limits live in /etc/security/limits.conf, enforced by PAM at login. Each line names a domain, a type, an item, and a value: appuser soft nofile 16384 sets the soft ceiling, appuser hard nofile 65536 sets the hard ceiling. The star domain sets defaults for everyone, but prefer named users or groups like @appteam so one change cannot surprise the whole fleet. Two gotchas bite every team once. First, limits.conf applies at session start through pam_limits: existing SSH sessions and running processes keep old values until re-login or restart, so always verify with a fresh login. Second, it does not touch systemd services at all: daemons launched by PID 1 never pass through PAM login, so their limits come from the unit file instead. Syntax errors here fail silently at login, leaving defaults in place with no warning, which is why some raises mysteriously do nothing. Check /etc/pam.d for session required pam_limits.so on custom stacks, keep one root session open while editing, and confirm with ulimit -n after a fresh login before closing out. Store the file in version control or a config manager so diffs show who changed what. Hand-edited hosts drift, and drift is how identical fleets develop mystery differences.

grep -R pam_limits /etc/pam.d/
cat /etc/security/limits.conf | grep -v '^#' | grep -v '^$'
su - appuser -c 'ulimit -Sn; ulimit -Hn'
cat /proc/1234/limits | grep -i open

systemd LimitNOFILE - The Override Teams Always Miss

On modern distros systemd owns service limits, and it ignores both shell ulimit and limits.conf for units. A service that needs 16,384 descriptors gets them only from LimitNOFILE in its unit. Check the live value with systemctl show app.service -p LimitNOFILE and the process truth with cat /proc/PID/limits. When those two disagree with your terminal, the unit wins and your shell work was theater. Set it with an override, never by editing vendor units: systemctl edit app.service writes /etc/systemd/system/app.service.d/override.conf where you put LimitNOFILE=16384, then systemctl daemon-reload and systemctl restart. Keep LimitNOFILEInfinity out of production; an uncapped service can exhaust the shared file-max and take neighbors down. Size from data: peak sockets plus files plus 30 percent headroom, rounded up. After restart, confirm the new ceiling in /proc/PID/limits and graph it. If you run containers under systemd, remember Docker and kubelet pass their own defaults down, so check the container runtime docs alongside the unit. Document the chosen number and its math in the unit comment. Add a deploy check that fails if LimitNOFILE is absent from the override. Explicit gates beat tribal knowledge every time the team grows.

systemctl show app.service -p LimitNOFILE
systemctl edit app.service
systemctl daemon-reload
systemctl restart app.service
cat /proc/1234/limits | grep -i open

Hunt the Leak With lsof - Find Who Holds What

When counts climb without extra traffic, something opens and never closes. lsof -p PID lists every descriptor with its type and target: REG for files, IPv4 or TCP for sockets, FIFO for pipes. Rank suspects across the box with an awk over the PID column, then zoom into the top holder. Read the NAME column like a detective. Hundreds of lines to one log path mean rotation is missing or the app holds deleted files open. Thousands of socket lines to one peer, especially in CLOSE_WAIT, mean retry or close logic is broken. Deleted markers mean disk space will not free until the holder restarts. Confirm with /proc/PID/fd directly: ls -l shows live symlinks, and sampling twice ten minutes apart proves growth versus steady state. For deeper proof, strace -e trace=open,openat,close -p PID for a minute shows opens without matching closes. Fix the code path, not the number: add missing close calls, use try-with-resources or finally blocks, enable log rotation, and cap connection pools. Re-run the same lsof after deploy and demand a flat line under load before closing the ticket. Save the before and after lsof summaries in the incident note. Concrete counts convince reviewers faster than claims that the leak is gone.

lsof -p 1234 | wc -l
lsof -p 1234 | grep -c CLOSE_WAIT
ls -l /proc/1234/fd | head -30
lsof -p 1234 | awk '{print $NF}' | sort | uniq -c | sort -rn | head

Sockets Count Too - Size for Connections, Not Just Files

Teams size for log files and forget each TCP connection costs a descriptor too. A server with 900 keepalive connections, 200 open files, and a few pipes sits near 1,100 against a 1,024 cap and fails on the next accept. The error still says Too many open files, which sends people hunting logs while sockets are the culprit. Measure both sides: ss -s gives host socket totals by state, lsof -p PID counts the process share, and /proc/PID/fd shows the combined pressure. TIME_WAIT and CLOSE_WAIT deserve special attention. TIME_WAIT is normal TCP hygiene after close and recycles on its own, but CLOSE_WAIT means your app never closed its end and those fds never recycle. Tune keepalive timeouts and pool sizes to match real concurrency, fix retry paths to close on every outcome, and only then size LimitNOFILE with headroom over measured peak. Monitor established plus CLOSE_WAIT per service and alert before the ceiling. Connection churn at peak is when leaks turn from slow drips into outages, so load-test with failure injection and watch the fd line stay flat. Put the socket-plus-file math in the capacity doc next to CPU and memory. Descriptors are capacity too, and they deserve the same review each quarter.