Mid-level 6 min · March 06, 2026

Linux File System — No Space Left on Device with Free Space

20% free disk but no space error? Inode exhaustion from /tmp cron job.

N
Naren · Founder
Plain-English first. Then code. Then the interview question.
About
 ● Production Incident 🔎 Debug Guide
Quick Answer
  • Single root (/) contains everything — no separate drives like C:\ or D:\.
  • Standard layout by FHS: /etc=config, /var=data, /home=users, /tmp=scratch.
  • Absolute paths start from /; relative paths depend on current directory.
  • Permissions: rwx per owner/group/others, octal notation (755, 644, 600).
  • Inodes store metadata; running out of inodes stops file creation even with free space.
  • Virtual filesystems /proc and /sys expose kernel data as files — zero disk used.
Plain-English First

Imagine your entire computer is a giant office building. The Linux file system is the floor plan of that building — it tells you exactly where every room is, what's stored in each room, and how to get from one room to another. Just like a building has a lobby at the ground floor and different departments on different floors, Linux has a single starting point (called root) and everything else branches out from there. There are no separate 'buildings' like C: or D: drives — it's one connected structure, top to bottom.

Every time you run a command in a Linux terminal, copy a file, or install software, the Linux file system is quietly doing the heavy lifting behind the scenes. It's the invisible backbone of every Linux server, every Docker container, every cloud VM you'll ever touch as a DevOps engineer. Understanding it isn't optional — it's the foundation everything else is built on.

Before Linux, different operating systems stored files in completely different ways with no agreed standard. This made software hard to port, hard to maintain, and easy to break. Linux solved this with the Filesystem Hierarchy Standard (FHS) — a clearly defined blueprint that says exactly where system files live, where user data goes, where temporary files are kept, and why. Every Linux distro you'll ever meet — Ubuntu, CentOS, Debian, Alpine — follows this same blueprint.

By the end of this article you'll be able to navigate any Linux system with confidence, explain what every major directory is for, read file paths without guessing, and answer the Linux file system questions that actually come up in DevOps interviews. No previous Linux experience needed — we're starting from zero.

The Root of Everything — How the Linux File System Tree Works

In Windows you might be used to drives like C:\ and D:\. Linux throws that idea out entirely. Instead, everything — and we mean everything — lives under one single top-level directory called root, written as just a forward slash: /

Think of it like a family tree. The great-grandparent at the very top is /. Every single file, folder, device, and process on the system hangs off a branch below it. There is no 'outside' of this tree.

This matters because it makes the system predictable. No matter which Linux machine you sit down at — a tiny Raspberry Pi or a massive cloud server — the layout is the same. /etc always holds configuration files. /var always holds variable data like logs. /home always holds user files. Once you learn the map, you can navigate any Linux system on earth.

The technical term for this design is a hierarchical file system, but honestly just think of it as a tree of folders with / at the very top. Every path you type starts from there — either absolutely (starting with /) or relatively (starting from wherever you currently are).

explore_root_directory.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
#!/bin/bash
# Let's explore the very top of the Linux file system tree
# The 'ls' command lists what's inside a directory
# The '/' argument tells it to look at the root directory

ls /

# Now let's see it as a proper tree structure
# The 'tree' command shows directories visually (install with: sudo apt install tree)
# We use -L 1 to only go ONE level deep — otherwise it prints thousands of lines

tree -L 1 /

# Let's also see our current location in the file system
# 'pwd' stands for Print Working Directory — it tells you exactly where you are

pwd
Output
# Output of: ls /
bin boot dev etc home lib lib64 media mnt opt proc root run sbin srv sys tmp usr var
# Output of: tree -L 1 /
/
├── bin
├── boot
├── dev
├── etc
├── home
├── lib
├── lib64
├── media
├── mnt
├── opt
├── proc
├── root
├── run
├── sbin
├── srv
├── sys
├── tmp
├── usr
└── var
20 directories, 0 files
# Output of: pwd
/home/youruser
Root the directory vs root the user
Don't confuse '/' (the root directory, top of the file system tree) with '/root' (the home folder of the root superuser account). They're completely different things. '/' is where the entire system starts. '/root' is just the home folder for the admin user.
Production Insight
A Docker container's root filesystem is its own /
Containers share the host kernel but get their own mount namespace
If your app can't find a file, check it's inside the container's filesystem tree — not the host's
Key Takeaway
One tree from / to everywhere
No separate drive letters — everything is a descendant of /
This makes Linux layouts predictable across distros

Every Major Directory Explained — What Lives Where and Why

Here's where most beginner guides fail you — they list directories like a dictionary with no story. Let's actually understand each one by thinking about WHO put files there and WHY.

/bin holds essential binaries (programs) that every user needs even during early system startup — things like ls, cp, mv, and cat. Think of it as the essential tools drawer in your kitchen.

/etc (pronounced 'et-see') is the system's configuration cabinet. Every time you install software and it has settings, those settings live in /etc. Apache web server config? /etc/apache2. SSH settings? /etc/ssh/sshd_config. User accounts list? /etc/passwd.

/home is where real people live. Every user on the system gets their own sub-folder here — /home/alice, /home/bob. It's where your documents, downloads, and personal configs go.

/var holds variable data — stuff that changes constantly while the system runs. Logs are the big one: /var/log. Package manager data, mail spools, and database files live here too.

/tmp is a scratch pad. Files here are wiped on reboot. Never store anything important here.

/proc and /sys are virtual directories — they don't contain real files on disk. They're a live window into the Linux kernel. Reading a file in /proc actually asks the kernel for current system information in real time.

navigate_key_directories.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
#!/bin/bash
# Let's visit the most important directories and see what's actually in them

echo "=== Binaries in /bin ==="
# List a sample of the programs available to all users
ls /bin | head -20   # 'head -20' shows just the first 20 results

echo ""
echo "=== Configuration files in /etc ==="
# See what software has configuration here
ls /etc | head -20

echo ""
echo "=== User home directories in /home ==="
# Each user on the system has a folder here
ls /home

echo ""
echo "=== Recent system log entries from /var/log ==="
# The system constantly writes logs here — let's see the last 5 lines of the system log
tail -5 /var/log/syslog   # On CentOS/RHEL use: /var/log/messages

echo ""
echo "=== Live kernel data from /proc ==="
# This is NOT a real file — the kernel generates it on the fly when you read it
# It shows the current uptime of the system (how long it's been running)
cat /proc/uptime   # Two numbers: seconds uptime, seconds idle time

echo ""
echo "=== CPU info straight from the kernel ==="
# Again, not a real file — the kernel answers this query live
cat /proc/cpuinfo | grep 'model name' | head -2
Output
=== Binaries in /bin ===
bash
cat
chmod
chown
cp
date
dd
df
dir
echo
false
grep
gzip
hostname
kill
ln
ls
mkdir
mv
nano
=== Configuration files in /etc ===
apt
bash.bashrc
cron.d
crontab
default
environment
fstab
group
hostname
hosts
init.d
issue
kernelcrash
ldap
logrotate.conf
logrotate.d
lsb-release
mtab
network
nginx
=== User home directories in /home ===
alice bob deploy
=== Recent system log entries from /var/log ===
Jun 12 14:22:01 myserver CRON[3421]: (root) CMD (run-parts /etc/cron.hourly)
Jun 12 14:22:01 myserver CRON[3420]: (root) SESSION (open)
Jun 12 14:22:01 myserver CRON[3420]: (root) SESSION (close)
Jun 12 14:25:01 myserver CRON[3502]: (root) CMD (test -x /usr/sbin/anacron)
Jun 12 14:30:01 myserver CRON[3611]: (root) CMD (run-parts /etc/cron.hourly)
=== Live kernel data from /proc ==="
183426.52 712930.18
=== CPU info straight from the kernel ===
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
model name : Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz
Pro Tip: /proc is your debugging superpower
When a process is behaving weirdly, check /proc/[process-id]/ — it contains live info about every running process: its open files, memory maps, environment variables, and more. Try 'ls /proc/$$' to inspect your own current shell process right now.
Production Insight
/var fills up silently. Log rotation misconfiguration = disk full at 3 AM.
Monitor /var/log size separately — it's the #1 cause of unexpected disk failures.
Use logrotate with compression and retention policies.
Key Takeaway
/etc = config, /var = runtime data, /home = users
/tmp = scratch (wiped on boot)
/proc = live kernel window, not real files

Absolute vs Relative Paths — The GPS Coordinates of Your File System

Now that you know the layout of the city, you need to know how to give directions in it. In Linux, every file has an address called a path. There are two ways to express that address, and understanding both will save you from a lot of confusion.

An absolute path starts with / and gives the full address from the root of the system, no matter where you currently are. It's like a GPS coordinate — completely unambiguous. /home/alice/documents/report.txt will always find that file, whether you're in /tmp, /etc, or anywhere else.

A relative path starts from wherever you currently are (your working directory). If you're already inside /home/alice, you can just say documents/report.txt and Linux figures out the rest. Two dots (..) means 'go up one level'. One dot (.) means 'right here'.

Why does this matter? When you write shell scripts or Dockerfiles, using relative paths can cause scripts to break when run from a different directory. Absolute paths are bulletproof. In contrast, relative paths are faster to type interactively. Knowing which to use and when is a skill that separates competent Linux users from beginners.

paths_absolute_vs_relative.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
#!/bin/bash
# Demonstrating absolute vs relative paths with a practical example
# We'll create a small directory structure, then navigate it both ways

# First, create a demo folder structure in /tmp (safe scratch space)
mkdir -p /tmp/demo_project/src/utils   # -p creates all parent folders at once
mkdir -p /tmp/demo_project/config
touch /tmp/demo_project/src/main.py
touch /tmp/demo_project/config/settings.yml

echo "=== Directory structure created ==="
tree /tmp/demo_project

echo ""
echo "=== Navigating with ABSOLUTE paths ==="
# cd changes your current directory
# Absolute path — starts with / — works from ANYWHERE
cd /tmp/demo_project/src
pwd   # Confirm where we are

# List the config folder using its full absolute address
ls /tmp/demo_project/config   # Works no matter where we are

echo ""
echo "=== Navigating with RELATIVE paths ==="
# We're currently in /tmp/demo_project/src
# Go UP one level with .. then into config
cd ../config   # .. means 'one folder up', so this goes to /tmp/demo_project/config
pwd

# The single dot (.) means 'current directory'
ls .   # Same as: ls /tmp/demo_project/config

echo ""
echo "=== Using ~ as a shortcut for your home directory ==="
# The tilde character is a special shortcut — always means your home folder
echo "My home directory is: ~"
cd ~
pwd   # Will show /home/yourusername

# Clean up our demo
rm -rf /tmp/demo_project
echo "Demo cleaned up."
Output
=== Directory structure created ===
/tmp/demo_project
├── config
│ └── settings.yml
└── src
├── main.py
└── utils
3 directories, 2 files
=== Navigating with ABSOLUTE paths ===
/tmp/demo_project/src
settings.yml
=== Navigating with RELATIVE paths ===
/tmp/demo_project/config
settings.yml
=== Using ~ as a shortcut for your home directory ===
My home directory is: ~
/home/alice
Demo cleaned up.
Watch Out: Scripts that break because of relative paths
A shell script that uses a relative path like './config/settings.yml' will only work correctly if you run it from the exact directory it expects. The fix: use $(dirname "$0") to get the script's own directory and build absolute paths from there. Example: CONFIG_FILE="$(dirname "$0")/config/settings.yml"
Production Insight
Cron runs scripts with $HOME set to /root or current user's home
Using a relative path in a cron job = file not found at 3 AM
Rule: always use absolute paths in automated scripts, or set cd explicitly
Key Takeaway
Absolute paths start with / — bulletproof
Relative paths break when the working directory changes
In scripts, use absolute paths (or dirname "$0")

File Permissions — Who's Allowed to Touch What

The Linux file system isn't just about WHERE files are stored — it also controls WHO can access them. This is the permission system, and it's one of the most important security concepts in Linux.

Every file and directory has three types of permission: read (r), write (w), and execute (x). And those permissions are set separately for three groups of people: the owner (the user who created the file), the group (a team of users), and everyone else (the world).

When you run ls -l, you see a string like -rwxr-xr-- at the start of each line. That's 10 characters. The first is the file type (- for file, d for directory). The next three are owner permissions. The next three are group permissions. The last three are everyone else's permissions.

Permissions are also expressed as numbers — this is called octal notation. r=4, w=2, x=1. Add them together for each group. So rwx = 7, r-x = 5, r-- = 4. A permission of 755 means the owner can do everything (7), and everyone else can read and execute but not write (5). This comes up constantly in DevOps when deploying files or running scripts.

file_permissions_demo.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
#!/bin/bash
# Hands-on demo of Linux file permissions
# We'll create files, read their permissions, and change them

# Create a sample script file
cat > /tmp/deploy_script.sh << 'EOF'
#!/bin/bash
echo "Deploying application..."
EOF

echo "=== Default permissions after creation ==="
# ls -l shows the long format with permissions
ls -l /tmp/deploy_script.sh
# You'll see something like: -rw-r--r-- 1 alice alice 42 Jun 12 14:00 deploy_script.sh
# That means: owner can read+write, group can read, everyone can read

echo ""
echo "=== Trying to EXECUTE without execute permission ==="
/tmp/deploy_script.sh   # This will fail — no execute permission yet!

echo ""
echo "=== Adding execute permission with chmod ==="
# chmod changes permissions
# 755 = rwxr-xr-x: owner has full access, group and others can read+execute
chmod 755 /tmp/deploy_script.sh
ls -l /tmp/deploy_script.sh   # Confirm the change

echo ""
echo "=== Now we can execute it ==="
/tmp/deploy_script.sh   # Works now!

echo ""
echo "=== Making a file secret — only the owner can read it ==="
echo "db_password=SuperSecret123" > /tmp/database.conf
chmod 600 /tmp/database.conf   # rw------- : only owner can read or write
ls -l /tmp/database.conf

echo ""
echo "=== Breaking down what we see in ls -l ==="
# Let's use stat for a crystal-clear view
stat /tmp/deploy_script.sh

# Clean up
rm /tmp/deploy_script.sh /tmp/database.conf
Output
=== Default permissions after creation ===
-rw-r--r-- 1 alice alice 42 Jun 12 14:00 /tmp/deploy_script.sh
=== Trying to EXECUTE without execute permission ===
bash: /tmp/deploy_script.sh: Permission denied
=== Adding execute permission with chmod ===
-rwxr-xr-x 1 alice alice 42 Jun 12 14:00 /tmp/deploy_script.sh
=== Now we can execute it ===
Deploying application...
=== Making a file secret — only the owner can read it ===
-rw------- 1 alice alice 26 Jun 12 14:00 /tmp/database.conf
=== Breaking down what we see in ls -l ===
File: /tmp/deploy_script.sh
Size: 42 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 1572867 Links: 1
Access: (0755/-rwxr-xr-x) Uid: ( 1000/ alice) Gid: ( 1000/ alice)
Access: 2024-06-12 14:00:01.000000000 +0000
Modify: 2024-06-12 14:00:01.000000000 +0000
Change: 2024-06-12 14:00:05.000000000 +0000
Birth: -
Interview Gold: The chmod 777 red flag
Interviewers love asking why chmod 777 is dangerous. The answer: it gives everyone on the system — including attackers who gain any foothold — full read, write, and execute access to that file. Never use 777 in production. Use the minimum permissions needed: 755 for executables, 644 for readable config files, 600 for secrets.
Production Insight
chmod 777 on a web directory lets Apache write but also lets any user deface the site.
If a scanner finds a 777 file, assume compromise.
Fix: use minimal permissions — 755 for directories, 644 for static files, 640 for configs owned by www-data.
Key Takeaway
rwx digits: 4=read, 2=write, 1=execute
Three groups: owner, group, others
Always apply least privilege — never 777 in production

Inodes and File Metadata — The Hidden Data Behind Every File

Every file on a Linux filesystem is tracked by an inode — a data structure that stores everything about the file except its name. Think of it as the file's passport: it knows the file's size, owner, permissions, timestamps, and which disk blocks hold the actual data. But surprisingly, the file's name isn't in the inode — that lives in a directory entry.

This separation matters. When you move a file within the same filesystem, only the directory entry changes. The inode and data blocks stay put. That's why mv is nearly instant on the same partition but slow across partitions (where data must be copied).

Inodes are a finite resource. When you create a filesystem, a fixed number of inodes is allocated. If you create millions of tiny files, you can exhaust your inode pool even with plenty of disk space free. That's the 'No space left on device' error that df -h doesn't show.

To see inode usage: df -i. This is one of the first checks senior engineers run when a disk full error doesn't make sense.

inode_inspection.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
#!/bin/bash
# Checking inode usage and details about files

echo "=== Inode usage on root filesystem ==="
df -i /

echo ""
echo "=== Find the inode number of a file ==="
ls -li /etc/hosts   # -i prints inode number

echo ""
echo "=== Get all metadata about a file ==="
stat /etc/hosts   # Shows inode, permissions, timestamps, blocks

echo ""
echo "=== Find files with most inodes used in /var ==="
sudo find /var -xdev -type f | wc -l   # Counts files (each file = one inode)

echo ""
echo "=== Simulate a full inode scenario (don't run in production!) ==="
echo "# To reproduce in a test environment:"
echo "# Create a 100MB file system with few inodes:"
echo "dd if=/dev/zero of=/tmp/testfs bs=1M count=100"
echo "mkfs.ext4 -N 1000 /tmp/testfs   # Only 1000 inodes"
echo "mount -o loop /tmp/testfs /mnt"
echo "for i in {1..2000}; do touch /mnt/file_$i; done"
echo "# After 1000 files, you'll see 'No space left on device' with df -h showing free space"
Output
=== Inode usage on root filesystem ===
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda1 2,000,000 500,000 1,500,000 25% /
=== Find the inode number of a file ===
1441793 -rw-r--r-- 1 root root 321 Mar 22 10:15 /etc/hosts
=== Get all metadata about a file ===
File: /etc/hosts
Size: 321 Blocks: 8 IO Block: 4096 regular file
Device: fd01h/64769d Inode: 1441793 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2026-04-20 12:00:00.000000000 +0000
Modify: 2026-04-10 08:30:00.000000000 +0000
Change: 2026-04-10 08:30:00.000000000 +0000
Birth: -
=== Find files with most inodes used in /var ===
94872
Production Insight
Inode exhaustion is the silent killer. A cron job writing 100,000 tiny log files per hour can exhaust inodes in a day.
Prevention: use XFS (dynamic inodes) or ext4 with larger inode ratio.
Monitoring: add alert for df -i utilization > 80%.
Key Takeaway
Inodes store metadata, not names
df -i checks inode usage (df -h checks block usage)
Exhaust inodes = no new files, even with free space

Mounting — How Disks and Directors Become Part of the Tree

A new hard drive or USB stick doesn't automatically become part of the Linux file system. You have to mount it — attach it at a specific directory in the tree. That directory becomes the mount point, and everything inside it then shows the contents of the mounted device.

Mounting makes the tree dynamic. A 500GB data disk gets mounted at /mnt/data. A NAS share gets mounted at /mnt/backups. Even the root filesystem itself is mounted at / during boot by the kernel.

The key file is /etc/fstab — the filesystem table. It lists every device and its mount point, filesystem type, and mount options. This is what the system reads at boot to mount everything automatically.

Common mount options include: ro (read-only), noexec (prevent execution of binaries), nosuid (ignore setuid bits), and defaults (rw, suid, dev, exec, auto, nouser, async). Senior engineers use these to harden systems — for example, mounting /tmp with noexec prevents attackers from running downloaded scripts directly.

mount_demo.shBASH
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
#!/bin/bash
# Understanding mount points and the /etc/fstab file

echo "=== Currently mounted filesystems ==="
mount | grep -E '^/dev' | head -10

echo ""
echo "=== Disk space usage with mount points ==="
df -h | head -15

echo ""
echo "=== The fstab file — automatic mounts at boot ==="
cat /etc/fstab

echo ""
echo "=== Mount a loop device (safe demo with a file) ==="
echo "Creating a 50MB file and mounting it as a filesystem..."
dd if=/dev/zero of=/tmp/loop_demo.img bs=1M count=50 &> /dev/null
mkfs.ext4 -F /tmp/loop_demo.img &> /dev/null
mkdir -p /mnt/loop_demo
sudo mount -o loop /tmp/loop_demo.img /mnt/loop_demo
df -h /mnt/loop_demo

echo ""
echo "=== Write a file to the mounted filesystem ==="
echo "Hello from loop device" | sudo tee /mnt/loop_demo/hello.txt
echo "Content:"
cat /mnt/loop_demo/hello.txt

echo ""
echo "=== Unmount and clean up ==="
sudo umount /mnt/loop_demo
rm /tmp/loop_demo.img
rmdir /mnt/loop_demo
echo "Done."
Output
=== Currently mounted filesystems ===
/dev/sda1 on / type ext4 (rw,relatime,errors=remount-ro)
/dev/sdb1 on /mnt/data type xfs (rw,relatime,attr2,inode64,noquota)
=== Disk space usage with mount points ===
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 20G 15G 5.0G 75% /
/dev/sdb1 500G 200G 300G 40% /mnt/data
=== The fstab file — automatic mounts at boot ===
UUID=abcde-1234 / ext4 defaults 0 1
UUID=12345-6789 /mnt/data xfs defaults 0 2
/dev/sdc1 /mnt/backup ext4 rw,noexec 0 2
192.168.1.100:/share /mnt/nfs nfs defaults,_netdev 0 0
=== Mount a loop device (safe demo with a file) ===
Filesystem Size Used Avail Use% Mounted on
/tmp/loop_demo.img 47M 12K 45M 1% /mnt/loop_demo
=== Write a file to the mounted filesystem ===
Hello from loop device
=== Unmount and clean up ===
Done.
Production shortcut: findmnt
Instead of parsing 'mount' output, use 'findmnt' — it shows the tree hierarchy of mount points, their source devices, options, and which are read-only. Run 'findmnt -l' for a list, or 'findmnt -T /path' to see which mount point a path belongs to.
Production Insight
A noexec mount on /tmp stops attackers from running downloaded payloads — but also breaks legitimate software that installs into /tmp.
Measure the trade-off: if your CI runner uses /tmp for build artifacts, noexec will fail builds silently.
Also: mounting with options 'noatime' reduces disk writes significantly on busy servers.
Key Takeaway
Mounting attaches a device to a directory in the tree
/etc/fstab controls auto-mount options at boot
noexec, nosuid, ro are security hardening options
● Production incidentPOST-MORTEMseverity: high

Disk Full Alerts With 20% Free Space

Symptom
Applications fail with 'No space left on device'. 'df -h' shows plenty of free space (20%+). 'df -i' reveals 100% inode usage.
Assumption
The team assumed free disk space meant free space to write files. They ignored inode monitoring.
Root cause
A runaway cron job created millions of tiny empty files in /tmp. Each file consumes one inode. With a limited inode pool (typically 1–2 million on a 20GB partition), the filesystem ran out of inodes before running out of disk blocks.
Fix
Deleted the excess files: 'find /tmp -type f -empty -delete'. Added inode monitoring to alert at 80% usage. Switched to a filesystem with dynamic inode allocation (XFS) on new partitions.
Key lesson
  • Always monitor both disk space and inode usage — df -h and df -i are equally important.
  • Small-file-heavy workloads (caches, logs, temp) exhaust inodes fast.
  • Choose XFS or ext4 with larger inode ratio for directories with millions of tiny files.
Production debug guideSymptom to action — the real commands that diagnose file system issues5 entries
Symptom · 01
Cannot create file: 'No space left on device'
Fix
Run 'df -h' for disk space, 'df -i' for inodes. 'du -sh /*' to find large directories.
Symptom · 02
Permission denied on a file you should own
Fix
Check owner with 'ls -l', compare against 'id'. Use 'stat' for detailed permissions. 'getfacl' if ACLs are set.
Symptom · 03
File is read-only but you're root
Fix
Check filesystem mount options with 'mount | grep <path>' — likely 'ro' flag. Remount with 'mount -o remount,rw'.
Symptom · 04
Slow directory listing in directory with millions of files
Fix
Avoid 'ls -l' — use 'ls -Uf' or 'find -maxdepth 1 -type f'. Consider splitting files into subdirectories (e.g., by date).
Symptom · 05
Disk I/O at 100% but no obvious culprit
Fix
Run 'iotop' or 'iostat -x 1'. Check 'lsof' for deleted but still-open files — they still consume disk space until closed.
★ Quick Debug Cheat Sheet: File System FullWhen your app says 'No space left' — follow these 5 steps in order.
No space left on device error
Immediate action
Stop all writes to the affected mount point.
Commands
df -h <mount_point> # Check disk usage
df -i <mount_point> # Check inode usage
Fix now
Delete oldest logs: sudo find <path> -name '.log' -mtime +30 -delete. Or compress: gzip .log
Space free but still getting 'no space'+
Immediate action
Check for deleted but still-open files.
Commands
lsof +L1 # Show all files with link count 0 (deleted but held open)
lsof -nP | grep '(deleted)' | awk '{print $2}' | sort -u
Fix now
Restart the process holding the deleted file: 'sudo systemctl restart <service>'
Inode 100% but space free+
Immediate action
Find the directory with excessive small files.
Commands
for d in /*; do echo -n "$d: "; find "$d" -xdev | wc -l; done # Count files per top-level dir
find /tmp -type f -atime +1 -delete # Example: delete old temp files
Fix now
Create a new partition with larger inode ratio: mkfs.ext4 -i 4096 /dev/sdb1 # 4096 bytes per inode
Linux Directory Structure Reference
DirectoryPurposeTypical ContentsWho Writes Here
/binEssential user binariesls, cp, mv, cat, bashOS installer
/etcSystem-wide configurationnginx.conf, sshd_config, fstabAdmins & package managers
/homeUser personal filesDocuments, dotfiles, downloadsIndividual users
/varVariable runtime dataLogs, caches, mail spools, databasesRunning services & OS
/tmpTemporary scratch spacePartial downloads, build artifactsAny process — wiped on reboot
/usrUser programs & librariesMost installed software and libsPackage managers
/optOptional third-party softwareManually installed apps, vendorsAdmins & vendors
/procLive kernel data (virtual)CPU info, process info, uptimeLinux kernel (no disk space used)
/devDevice filessda (disk), tty (terminal), null, zeroLinux kernel
/bootBoot filesKernel image, GRUB bootloader configOS installer & kernel updates

Key takeaways

1
Everything in Linux lives under a single root directory '/'
there are no separate drives like Windows C:\ or D:\, just one connected tree of directories branching from that single point.
2
The directory layout is standardised by the Filesystem Hierarchy Standard (FHS)
/etc is always config, /var is always runtime data, /home is always user files — so once you learn the map, every Linux system on earth makes sense.
3
Absolute paths (starting with /) are bulletproof and always work regardless of where you are in the filesystem
always use them in scripts and cron jobs to avoid hard-to-debug failures.
4
File permissions control read (r=4), write (w=2), and execute (x=1) separately for the owner, group, and everyone else
understanding octal notation like 755 and 600 is essential for secure DevOps work.
5
Inodes are a finite resource
monitor both df -h and df -i to catch disk-full issues before they crash applications.
6
Mounting attaches storage at a specific directory
use /etc/fstab with UUIDs for permanent mounts, and security options like noexec and nosuid to harden partitions.

Common mistakes to avoid

5 patterns
×

Confusing /root with /

Symptom
Typing 'cd root' takes you to /home/youruser/root (which doesn't exist) or you get 'No such file or directory'. You think the admin home folder is the top of the filesystem.
Fix
Remember: '/' is the top of the tree. '/root' (with a leading slash) is the home directory of the root user. Always use 'cd /' to go to the absolute top.
×

Using relative paths in cron jobs or scripts

Symptom
Your script works perfectly when you run it manually but silently fails in a cron job — can't find config files, output goes to the wrong place.
Fix
Cron jobs run with / as the working directory. Always use absolute paths in scripts that run automatically, or set the working directory explicitly: 'cd /path/to/your/app || exit 1' at the top of the script.
×

Running chmod 777 to 'fix' permission errors

Symptom
The permission denied error goes away, but now the file is writable by every process and user on the system, including any attacker who gains the lowest-privilege access.
Fix
Think about WHO actually needs access. A web server config file that nginx reads should be 644 (owner writes, everyone reads). A credentials file should be 600 (only owner reads/writes). Start restrictive and loosen only as needed.
×

Ignoring inode usage when disk space looks fine

Symptom
Application fails with 'No space left on device' but 'df -h' shows 30% free space. You're stuck for a while before finding the issue.
Fix
Always run 'df -i' when you see a 'no space' error. Monitor inode usage with the same priority as disk space — especially on partitions that receive many small files (caches, logs, temp).
×

Mounting a new drive without checking fstab for boot

Symptom
You manually mount a drive, write data to it, then reboot. The data is there but the mount point is empty — the drive didn't mount automatically.
Fix
Add an entry to /etc/fstab for permanent mounts. Use filesystem UUIDs (blkid to get them) instead of device names like /dev/sdb1, which can change on reboot. Always test with 'mount -a' after editing fstab.
INTERVIEW PREP · PRACTICE MODE

Interview Questions on This Topic

Q01SENIOR
Walk me through the Linux file system hierarchy. What's the difference b...
Q02SENIOR
What's an inode in the Linux file system, and why would you care if a di...
Q03SENIOR
A junior dev on your team ran 'chmod 777 -R /var/www/html' to fix a web ...
Q04SENIOR
Explain the difference between a hard link and a symbolic link in Linux....
Q01 of 04SENIOR

Walk me through the Linux file system hierarchy. What's the difference between /bin, /usr/bin, and /usr/local/bin — and why do all three exist?

ANSWER
/bin contains essential binaries needed for booting and recovery, even if /usr isn't mounted. /usr/bin holds most user commands (it's mounted separately in some historical setups). /usr/local/bin is for locally compiled/installed software — it's intentionally separate so package manager updates don't overwrite custom tools. In modern distros /bin is often a symlink to /usr/bin, but the separation originated from mounting /usr over the network.
FAQ · 5 QUESTIONS

Frequently Asked Questions

01
What is the Linux file system and how is it different from Windows?
02
What is /etc in Linux and what does it stand for?
03
Why can't I find files in /proc on disk — are they real files?
04
What happens if I run out of inodes on a filesystem?
05
How do I mount a USB drive in Linux?
🔥

That's Linux. Mark it forged?

6 min read · try the examples if you haven't

Previous
Linux Command Line Basics
2 / 12 · Linux
Next
Linux File Permissions