Linux File System — No Space Left on Device with Free Space

Every time you run a command in a Linux terminal, copy a file, or install software, the Linux file system is quietly doing the heavy lifting behind the scenes. It's the invisible backbone of every Linux server, every Docker container, every cloud VM you'll ever touch as a DevOps engineer. Understanding it isn't optional — it's the foundation everything else is built on.

Before Linux, different operating systems stored files in completely different ways with no agreed standard. This made software hard to port, hard to maintain, and easy to break. Linux solved this with the Filesystem Hierarchy Standard (FHS) — a clearly defined blueprint that says exactly where system files live, where user data goes, where temporary files are kept, and why. Every Linux distro you'll ever meet — Ubuntu, CentOS, Debian, Alpine — follows this same blueprint.

By the end of this article you'll be able to navigate any Linux system with confidence, explain what every major directory is for, read file paths without guessing, and answer the Linux file system questions that actually come up in DevOps interviews. No previous Linux experience needed — we're starting from zero.

What the Linux File System Actually Is

The Linux file system is a hierarchical namespace rooted at '/' that maps human-readable paths to inodes — metadata structures storing file attributes and disk block pointers. The core mechanic is the separation of directory entries (names) from inodes (data), enabling hard links and atomic renames. This design is what makes 'No space left on device' possible even when 'df' shows free space: inode exhaustion or filesystem metadata corruption can block writes independently of data blocks.

In practice, the VFS (Virtual File System) abstraction lets ext4, XFS, Btrfs, and others coexist under a single syscall interface. Key properties: block allocation strategies (extents vs. bitmaps), journaling guarantees (ordered, writeback, data), and reserved blocks for root (5% by default on ext4). These directly impact write latency, crash recovery, and the 'disk full' threshold. A 1% reserved block tweak on a 10TB volume reclaims 100GB — but breaks emergency root access.

Use this knowledge when diagnosing 'disk full' alerts: always check both 'df -h' (block usage) and 'df -i' (inode usage). In containerized or high-file-count workloads (e.g., message queues, build caches), inode exhaustion is the silent killer. Understanding the filesystem's internal accounting prevents false positives and wasted debugging cycles.

The Root of Everything — How the Linux File System Tree Works

In Windows you might be used to drives like C:\ and D:\. Linux throws that idea out entirely. Instead, everything — and we mean everything — lives under one single top-level directory called root, written as just a forward slash: /

Think of it like a family tree. The great-grandparent at the very top is /. Every single file, folder, device, and process on the system hangs off a branch below it. There is no 'outside' of this tree.

This matters because it makes the system predictable. No matter which Linux machine you sit down at — a tiny Raspberry Pi or a massive cloud server — the layout is the same. /etc always holds configuration files. /var always holds variable data like logs. /home always holds user files. Once you learn the map, you can navigate any Linux system on earth.

The technical term for this design is a hierarchical file system, but honestly just think of it as a tree of folders with / at the very top. Every path you type starts from there — either absolutely (starting with /) or relatively (starting from wherever you currently are).

#!/bin/bash
# Let's explore the very top of the Linux file system tree
# The 'ls' command lists what's inside a directory
# The '/' argument tells it to look at the root directory

ls /

# Now let's see it as a proper tree structure
# The 'tree' command shows directories visually (install with: sudo apt install tree)
# We use -L 1 to only go ONE level deep — otherwise it prints thousands of lines

tree -L 1 /

# Let's also see our current location in the file system
# 'pwd' stands for Print Working Directory — it tells you exactly where you are

pwd

Every Major Directory Explained — What Lives Where and Why

Here's where most beginner guides fail you — they list directories like a dictionary with no story. Let's actually understand each one by thinking about WHO put files there and WHY.

/bin holds essential binaries (programs) that every user needs even during early system startup — things like ls, cp, mv, and cat. Think of it as the essential tools drawer in your kitchen.

/etc (pronounced 'et-see') is the system's configuration cabinet. Every time you install software and it has settings, those settings live in /etc. Apache web server config? /etc/apache2. SSH settings? /etc/ssh/sshd_config. User accounts list? /etc/passwd.

/home is where real people live. Every user on the system gets their own sub-folder here — /home/alice, /home/bob. It's where your documents, downloads, and personal configs go.

/var holds variable data — stuff that changes constantly while the system runs. Logs are the big one: /var/log. Package manager data, mail spools, and database files live here too.

/tmp is a scratch pad. Files here are wiped on reboot. Never store anything important here.

/proc and /sys are virtual directories — they don't contain real files on disk. They're a live window into the Linux kernel. Reading a file in /proc actually asks the kernel for current system information in real time.

#!/bin/bash
# Let's visit the most important directories and see what's actually in them

echo "=== Binaries in /bin ==="
# List a sample of the programs available to all users
ls /bin | head -20   # 'head -20' shows just the first 20 results

echo ""
echo "=== Configuration files in /etc ==="
# See what software has configuration here
ls /etc | head -20

echo ""
echo "=== User home directories in /home ==="
# Each user on the system has a folder here
ls /home

echo ""
echo "=== Recent system log entries from /var/log ==="
# The system constantly writes logs here — let's see the last 5 lines of the system log
tail -5 /var/log/syslog   # On CentOS/RHEL use: /var/log/messages

echo ""
echo "=== Live kernel data from /proc ==="
# This is NOT a real file — the kernel generates it on the fly when you read it
# It shows the current uptime of the system (how long it's been running)
cat /proc/uptime   # Two numbers: seconds uptime, seconds idle time

echo ""
echo "=== CPU info straight from the kernel ==="
# Again, not a real file — the kernel answers this query live
cat /proc/cpuinfo | grep 'model name' | head -2

Absolute vs Relative Paths — The GPS Coordinates of Your File System

Now that you know the layout of the city, you need to know how to give directions in it. In Linux, every file has an address called a path. There are two ways to express that address, and understanding both will save you from a lot of confusion.

An absolute path starts with / and gives the full address from the root of the system, no matter where you currently are. It's like a GPS coordinate — completely unambiguous. /home/alice/documents/report.txt will always find that file, whether you're in /tmp, /etc, or anywhere else.

A relative path starts from wherever you currently are (your working directory). If you're already inside /home/alice, you can just say documents/report.txt and Linux figures out the rest. Two dots (..) means 'go up one level'. One dot (.) means 'right here'.

Why does this matter? When you write shell scripts or Dockerfiles, using relative paths can cause scripts to break when run from a different directory. Absolute paths are bulletproof. In contrast, relative paths are faster to type interactively. Knowing which to use and when is a skill that separates competent Linux users from beginners.

#!/bin/bash
# Demonstrating absolute vs relative paths with a practical example
# We'll create a small directory structure, then navigate it both ways

# First, create a demo folder structure in /tmp (safe scratch space)
mkdir -p /tmp/demo_project/src/utils   # -p creates all parent folders at once
mkdir -p /tmp/demo_project/config
touch /tmp/demo_project/src/main.py
touch /tmp/demo_project/config/settings.yml

echo "=== Directory structure created ==="
tree /tmp/demo_project

echo ""
echo "=== Navigating with ABSOLUTE paths ==="
# cd changes your current directory
# Absolute path — starts with / — works from ANYWHERE
cd /tmp/demo_project/src
pwd   # Confirm where we are

# List the config folder using its full absolute address
ls /tmp/demo_project/config   # Works no matter where we are

echo ""
echo "=== Navigating with RELATIVE paths ==="
# We're currently in /tmp/demo_project/src
# Go UP one level with .. then into config
cd ../config   # .. means 'one folder up', so this goes to /tmp/demo_project/config
pwd

# The single dot (.) means 'current directory'
ls .   # Same as: ls /tmp/demo_project/config

echo ""
echo "=== Using ~ as a shortcut for your home directory ==="
# The tilde character is a special shortcut — always means your home folder
echo "My home directory is: ~"
cd ~
pwd   # Will show /home/yourusername

# Clean up our demo
rm -rf /tmp/demo_project
echo "Demo cleaned up."

File Permissions — Who's Allowed to Touch What

The Linux file system isn't just about WHERE files are stored — it also controls WHO can access them. This is the permission system, and it's one of the most important security concepts in Linux.

Every file and directory has three types of permission: read (r), write (w), and execute (x). And those permissions are set separately for three groups of people: the owner (the user who created the file), the group (a team of users), and everyone else (the world).

When you run ls -l, you see a string like -rwxr-xr-- at the start of each line. That's 10 characters. The first is the file type (- for file, d for directory). The next three are owner permissions. The next three are group permissions. The last three are everyone else's permissions.

Permissions are also expressed as numbers — this is called octal notation. r=4, w=2, x=1. Add them together for each group. So rwx = 7, r-x = 5, r-- = 4. A permission of 755 means the owner can do everything (7), and everyone else can read and execute but not write (5). This comes up constantly in DevOps when deploying files or running scripts.

#!/bin/bash
# Hands-on demo of Linux file permissions
# We'll create files, read their permissions, and change them

# Create a sample script file
cat > /tmp/deploy_script.sh << 'EOF'
#!/bin/bash
echo "Deploying application..."
EOF

echo "=== Default permissions after creation ==="
# ls -l shows the long format with permissions
ls -l /tmp/deploy_script.sh
# You'll see something like: -rw-r--r-- 1 alice alice 42 Jun 12 14:00 deploy_script.sh
# That means: owner can read+write, group can read, everyone can read

echo ""
echo "=== Trying to EXECUTE without execute permission ==="
/tmp/deploy_script.sh   # This will fail — no execute permission yet!

echo ""
echo "=== Adding execute permission with chmod ==="
# chmod changes permissions
# 755 = rwxr-xr-x: owner has full access, group and others can read+execute
chmod 755 /tmp/deploy_script.sh
ls -l /tmp/deploy_script.sh   # Confirm the change

echo ""
echo "=== Now we can execute it ==="
/tmp/deploy_script.sh   # Works now!

echo ""
echo "=== Making a file secret — only the owner can read it ==="
echo "db_password=SuperSecret123" > /tmp/database.conf
chmod 600 /tmp/database.conf   # rw------- : only owner can read or write
ls -l /tmp/database.conf

echo ""
echo "=== Breaking down what we see in ls -l ==="
# Let's use stat for a crystal-clear view
stat /tmp/deploy_script.sh

# Clean up
rm /tmp/deploy_script.sh /tmp/database.conf

Inodes and File Metadata — The Hidden Data Behind Every File

Every file on a Linux filesystem is tracked by an inode — a data structure that stores everything about the file except its name. Think of it as the file's passport: it knows the file's size, owner, permissions, timestamps, and which disk blocks hold the actual data. But surprisingly, the file's name isn't in the inode — that lives in a directory entry.

This separation matters. When you move a file within the same filesystem, only the directory entry changes. The inode and data blocks stay put. That's why mv is nearly instant on the same partition but slow across partitions (where data must be copied).

Inodes are a finite resource. When you create a filesystem, a fixed number of inodes is allocated. If you create millions of tiny files, you can exhaust your inode pool even with plenty of disk space free. That's the 'No space left on device' error that df -h doesn't show.

To see inode usage: df -i. This is one of the first checks senior engineers run when a disk full error doesn't make sense.

#!/bin/bash
# Checking inode usage and details about files

echo "=== Inode usage on root filesystem ==="
df -i /

echo ""
echo "=== Find the inode number of a file ==="
ls -li /etc/hosts   # -i prints inode number

echo ""
echo "=== Get all metadata about a file ==="
stat /etc/hosts   # Shows inode, permissions, timestamps, blocks

echo ""
echo "=== Find files with most inodes used in /var ==="
sudo find /var -xdev -type f | wc -l   # Counts files (each file = one inode)

echo ""
echo "=== Simulate a full inode scenario (don't run in production!) ==="
echo "# To reproduce in a test environment:"
echo "# Create a 100MB file system with few inodes:"
echo "dd if=/dev/zero of=/tmp/testfs bs=1M count=100"
echo "mkfs.ext4 -N 1000 /tmp/testfs   # Only 1000 inodes"
echo "mount -o loop /tmp/testfs /mnt"
echo "for i in {1..2000}; do touch /mnt/file_$i; done"
echo "# After 1000 files, you'll see 'No space left on device' with df -h showing free space"

Mounting — How Disks and Directors Become Part of the Tree

A new hard drive or USB stick doesn't automatically become part of the Linux file system. You have to mount it — attach it at a specific directory in the tree. That directory becomes the mount point, and everything inside it then shows the contents of the mounted device.

Mounting makes the tree dynamic. A 500GB data disk gets mounted at /mnt/data. A NAS share gets mounted at /mnt/backups. Even the root filesystem itself is mounted at / during boot by the kernel.

The key file is /etc/fstab — the filesystem table. It lists every device and its mount point, filesystem type, and mount options. This is what the system reads at boot to mount everything automatically.

Common mount options include: ro (read-only), noexec (prevent execution of binaries), nosuid (ignore setuid bits), and defaults (rw, suid, dev, exec, auto, nouser, async). Senior engineers use these to harden systems — for example, mounting /tmp with noexec prevents attackers from running downloaded scripts directly.

#!/bin/bash
# Understanding mount points and the /etc/fstab file

echo "=== Currently mounted filesystems ==="
mount | grep -E '^/dev' | head -10

echo ""
echo "=== Disk space usage with mount points ==="
df -h | head -15

echo ""
echo "=== The fstab file — automatic mounts at boot ==="
cat /etc/fstab

echo ""
echo "=== Mount a loop device (safe demo with a file) ==="
echo "Creating a 50MB file and mounting it as a filesystem..."
dd if=/dev/zero of=/tmp/loop_demo.img bs=1M count=50 &> /dev/null
mkfs.ext4 -F /tmp/loop_demo.img &> /dev/null
mkdir -p /mnt/loop_demo
sudo mount -o loop /tmp/loop_demo.img /mnt/loop_demo
df -h /mnt/loop_demo

echo ""
echo "=== Write a file to the mounted filesystem ==="
echo "Hello from loop device" | sudo tee /mnt/loop_demo/hello.txt
echo "Content:"
cat /mnt/loop_demo/hello.txt

echo ""
echo "=== Unmount and clean up ==="
sudo umount /mnt/loop_demo
rm /tmp/loop_demo.img
rmdir /mnt/loop_demo
echo "Done."

File Operations That Don't Bite Back — Soft Links vs Hard Links

You've seen ln in scripts. You've probably used it wrong. Links aren't magic — they're just directory entries pointing to inodes. A hard link is a second name for the same inode. Delete one, the other survives. Soft links (symlinks) are special files containing a path string. Break the target, the link dangles.

Why this matters in production: hard links can't cross filesystem boundaries. A symlink can, but if you rsync a symlink without --copy-links, you'll copy a broken pointer. I've seen a deployment pipeline silently lose SSL certificates because someone symlinked /etc/ssl/certs into a container filesystem that didn't mount the host's /etc.

Use hard links for backup snapshots. Companies like rsync.net rely on them. Use symlinks for config management — but always check the target exists before the link does. readlink -f is your friend. Never use ln -s in a script without trapping the exit code.

// io.thecodeforge — devops tutorial

# Create a timestamped snapshot directory
- name: Create hard-linked backup snapshot
  ansible.builtin.shell: |
    SNAPSHOT_DIR="/backups/{{ inventory_hostname }}/$(date +%Y%m%d_%H%M%S)"
    cp -al /data/current/ "$SNAPSHOT_DIR"
    echo "Snapshot created at $SNAPSHOT_DIR"

# Verify inode sharing
- name: Show inodes of two files in snapshot
  ansible.builtin.stat:
    path: "{{ item }}"
  loop:
    - "/data/current/config.yml"
    - "/backups/{{ inventory_hostname }}/$(date +%Y%m%d)/config.yml"

Filesystem Types — When Ext4 Isn't Enough

Ext4 is the default. It's stable, journaled, and boring. Boring is good in production. But boring doesn't mean optimal. XFS excels with large files — think database dumps, video streams. Btrfs and ZFS give you snapshots, compression, and checksums. But they trade complexity for features.

Your choice should reflect your workload. Running MySQL? XFS on LVM gives you online resizing and consistent performance for sequential writes. Hosting containers on Docker? OverlayFS sits on top of any filesystem — but if you use Btrfs natively, Docker can use subvolumes for faster layer management.

Reality check: you'll hit a filesystem limit during an outage. Ext4 maxes out at 1 exabyte filesystem and 16 TB per file. XFS handles 8 exabytes. Btrfs can go to 16 exabytes. If you're managing petabyte-scale storage, ext4 will fail silently. Always df -T before blaming the disk.

// io.thecodeforge — devops tutorial

- name: Check filesystem type on mounted volumes
  hosts: production_db
  tasks:
    - name: Gather filesystem facts
      ansible.builtin.command: df -T --type=ext4 --type=xfs --type=btrfs
      register: fs_info

    - name: Show filesystem types
      ansible.builtin.debug:
        msg: "{{ fs_info.stdout_lines }}"

# Output will show:
# Filesystem     Type     1K-blocks    Used Available Use% Mounted on
# /dev/sda1      ext4     20511356 7349500  12121656  38% /
# /dev/sdb1      xfs      52403200 1123456  51279744   3% /data

Filesystem Types — When Ext4 Isn't Enough

Ext4 is the default for most Linux distributions, but it fails in specific workloads. Btrfs offers copy-on-write, snapshots, and built-in RAID – essential for containers and rollback scenarios. XFS excels with large files and parallel I/O, making it ideal for media servers and databases. ZFS (via OpenZFS) provides enterprise-grade features like checksumming, compression, and pool-based storage, but requires manual setup. Choose Btrfs for flexibility, XFS for throughput, and ZFS for data integrity. Mount a Btrfs subvolume for Docker storage to avoid overlay2 layer bloat. Use XFS with large stripe widths on RAID arrays. Avoid Ext4 when you need checksumming or atomic snapshots. Test filesystem performance with fio before production deployment.

// io.thecodeforge — devops tutorial

filesystem_choice:
  - use: btrfs
    when: snapshots, containers, rollback
    mount: /var/lib/docker
    command: mkfs.btrfs /dev/sdb
  - use: xfs
    when: large files, parallel writes
    mount: /data/media
    command: mkfs.xfs -d su=256k,sw=4 /dev/sdc
  - use: zfs
    when: data integrity, pools
    mount: /storage
    command: zpool create tank /dev/sdd

File Operations That Don't Bite Back — Soft Links vs Hard Links

Hard links create multiple directory entries pointing to the same inode. They share the same data blocks and permissions. Deleting one leaves the others intact. Soft links (symlinks) are pointers to a path – they break if the target moves or is deleted. Use hard links for deduplication within the same filesystem (they cannot cross mount points). Use soft links for version switching, cross-filesystem references, or directory links. Check hard link count with ls -l. A file with link count 2 exists in two places. Changing permissions on any hard link affects all. Soft links can chain, hard links cannot. Use stat to inspect inode numbers. Mistake: linking across filesystems causes hard link failures.

// io.thecodeforge — devops tutorial

# Hard link
- command: ln /data/file1 /backup/file1_hard
  effect: same inode, different name

# Soft link
- command: ln -s /data/file1 /backup/file1_soft
  effect: pointer, breaks if target moves

# Verify
- command: ls -li /data/file1 /backup/file1_*
  field: link_count
  example: "2" for hard link

# Cannot cross filesystem
- warning: "ln: failed to create hard link '/mnt/usb/': Invalid cross-device link"

Archiving, Compressing & Networking Services

Before you ship logs or deploy artifacts, you must understand why archiving and compression are separate steps. Archiving (tar) bundles files into one stream while preserving directory structure. Compression (gzip, bzip2, xz) reduces size. When you see .tar.gz, you’re combining both. Why this matters for DevOps: compressed archives reduce bandwidth and disk usage, but they also mask file-level metadata—always archive first to keep permissions and inodes intact. Networking services like sshd, nginx, and httpd rely on ports and sockets that are files under /proc and /sys. Understanding this link helps you debug why a service can’t start: check open file descriptors, not just process status. For background work, use nohup or disown to keep jobs alive after logout; otherwise, network services drop connections. A common trap: compressing a live log file while it’s being written corrupts the archive. Stop the writer first, then archive.

// io.thecodeforge — devops tutorial
// 25 lines max
- name: Archive app logs before rotation
  command: tar czf /backup/app-{{ ansible_date_time.epoch }}.tar.gz /var/log/app/
  tags: archive
- name: Restart nginx after archive
  systemd:
    name: nginx
    state: restarted
  when: archive_result.changed
- name: Verify port 80 is listening
  shell: ss -tlnp | grep :80
  register: port_check
- name: Alert if missing
  debug:
    msg: "nginx not listening — check SELinux or missing file caps"
  when: port_check.stdout == ""

Managing Jobs, systemd & Advanced Shell Scripting

Why separate foreground and background jobs? In terminal sessions, foreground jobs block your shell; background jobs (appended with &) let you run concurrent tasks. But background jobs are tied to the shell session—logout kills them. Use disown or nohup to detach. This surfaces a deeper principle: every job is a process, and systemd is the modern process supervisor. Systemd replaces init scripts and manages services as units. When you run systemctl start, you’re telling systemd to fork a process, track its PID, and restart on failure. Why this matters for scripting: advanced shell scripts must handle exit codes, traps, and exec to replace the shell process. A trap on SIGTERM ensures clean shutdown. For example, a script running a background websocket should trap SIGINT to kill children. Avoid subshells in loops—they create hidden processes. Use exec for long-running daemons to avoid zombie processes. A common footgun: forgetting that && and || have equal precedence in shell; always group with curly braces or parentheses. Systemd also provides environment files and templated units—use them instead of sourcing scripts in-line.

// io.thecodeforge — devops tutorial
// 25 lines max
- name: Create systemd service for health check
  template:
    src: health_check.service.j2
    dest: /etc/systemd/system/health_check.service
- name: Enable and start
  systemd:
    name: health_check
    enabled: yes
    state: started
- name: Check for zombies
  shell: ps aux | grep ' Z '
  register: zombies
- name: Alert on zombie processes
  debug:
    msg: "Zombie found: {{ zombies.stdout_lines }}"
  when: zombies.stdout != ""
- name: Kill runaway background jobs
  shell: "kill -9 {{ item }}"
  loop: "{{ zombie_pids }}"
  ignore_errors: yes