Linux Disk Management — The /dev/sdX Reboot Trap

Every production outage I've ever seen that started with 'disk' in the alert was caused by someone who treated storage as an afterthought. A full root partition kills web servers, a misconfigured filesystem destroys databases, and a missing mount point in /etc/fstab means your server reboots into chaos at 3 AM. Storage management isn't glamorous, but it is the difference between a system that hums along and one that pages you on a Friday night.

The problem is that most tutorials show you the commands and stop there. They'll tell you to run mkfs.ext4 without explaining that formatting is irreversible and takes seconds. They'll show you mount without mentioning it evaporates on reboot unless you wire it into /etc/fstab. The gap between 'ran the command in a tutorial' and 'confidently managing storage on a live server' is exactly where people get hurt.

By the end of this article you'll know how to inspect a disk from scratch, partition it intentionally, format it with the right filesystem for your workload, mount it persistently, and use LVM to manage storage dynamically when your needs change. These are the skills you actually need on the job — not just for passing an exam.

Inspecting What You Have — Reading the Disk Landscape Before Touching Anything

The first rule of storage management is: never run a destructive command on a disk you haven't fully inspected. This sounds obvious, but under pressure people confuse /dev/sda with /dev/sdb and wipe the wrong drive. It happens more than anyone admits.

lsblk is your safest starting point. It reads block device info from sysfs without touching the disk itself — no risk, no side effects. It shows you the full device tree: physical drives, their partitions, and any logical volumes sitting on top. fdisk -l goes deeper, showing partition types, sizes, and sector alignment, but it requires root.

df -h tells you about mounted filesystems — what's actually in use right now. Note the difference: lsblk shows you everything attached to the system, df -h shows only what's mounted and accessible. A disk can exist on lsblk and be completely invisible to df -h if nobody's mounted it yet. Understanding this distinction stops a whole class of 'where did my disk go?' confusion.

The UUID shown in blkid is critical — always use UUIDs in /etc/fstab, not device names like /dev/sdb1. Device names are assigned at boot time and can change if you add or remove hardware. UUIDs are permanent identifiers burned into the filesystem itself.

#!/bin/bash
# inspect_disk_landscape.sh
# Safe read-only commands to fully understand your storage before making any changes.
# Run as root (or with sudo) for full output.

echo "=== BLOCK DEVICE TREE (lsblk) ==="
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,UUID

echo ""
echo "=== PARTITION DETAILS (fdisk) ==="
sudo fdisk -l /dev/sda

echo ""
echo "=== MOUNTED FILESYSTEM USAGE (df) ==="
df -h --output=source,size,used,avail,pcent,target

echo ""
echo "=== FILESYSTEM UUIDs (blkid) ==="
sudo blkid

Partitioning, Formatting and Mounting — Preparing a New Disk From Scratch

When a fresh disk arrives — whether it's a new SSD in a bare-metal server or a new EBS volume attached to an EC2 instance — it's a blank slate. No partition table, no filesystem, no mount point. Before any application can write data to it, you need to walk through three distinct steps: partition, format, mount.

Partitioning with gdisk (for GPT) or fdisk (for MBR) defines the logical boundaries on the disk. For any disk over 2TB or any UEFI system, use GPT. For older systems or VMs where you know it's MBR, fdisk is fine. The partition table is just metadata that tells the OS where one region ends and another begins.

Formatting writes a filesystem into that partition. ext4 is the safe, well-understood default for general-purpose workloads — it has journaling, solid fsck tooling, and decades of battle testing. xfs is better for large files and high-throughput workloads (think log aggregation, big data). Don't overthink it for most use cases: ext4 unless you have a specific reason.

Mounting connects the formatted partition to a directory in the filesystem tree. The mount command does it immediately, but it vanishes on reboot. The /etc/fstab file makes it permanent. Every mounted filesystem you care about needs an entry there.

#!/bin/bash
# partition_format_mount.sh
# Full walkthrough: take a raw disk (/dev/sdb) and make it usable.
# WARNING: This DESTROYS all data on /dev/sdb. Verify the device name first.
# Prerequisites: run as root. Confirm target disk with: lsblk | grep sdb

TARGET_DISK="/dev/sdb"           # The raw disk we're preparing
PARTITION="/dev/sdb1"            # The partition we'll create
MOUNT_DIR="/mnt/appdata"         # Where we'll attach it in the filesystem tree
FS_LABEL="appdata-vol"           # Human-readable label (helpful in logs and blkid)

# --- STEP 1: PARTITION THE DISK ---
echo "Creating GPT partition table on ${TARGET_DISK}..."
sudo gdisk ${TARGET_DISK} <<EOF
n
1


8300
w
yes
EOF

sudo partprobe ${TARGET_DISK}
sleep 2
echo "Partition layout after gdisk:"
lsblk ${TARGET_DISK}

# --- STEP 2: FORMAT THE PARTITION ---
echo "Formatting ${PARTITION} as ext4..."
sudo mkfs.ext4 -L ${FS_LABEL} ${PARTITION}

# --- STEP 3: CREATE MOUNT POINT ---
sudo mkdir -p ${MOUNT_DIR}

# --- STEP 4: MOUNT TEMPORARILY (to verify it works) ---
sudo mount ${PARTITION} ${MOUNT_DIR}
echo "Temporary mount successful. Testing write access..."
echo "storage_test" | sudo tee ${MOUNT_DIR}/write_test.txt > /dev/null

# --- STEP 5: GET UUID FOR FSTAB ---
DISK_UUID=$(sudo blkid -s UUID -o value ${PARTITION})
echo "UUID for ${PARTITION}: ${DISK_UUID}"

# --- STEP 6: ADD TO /etc/fstab FOR PERSISTENT MOUNTING ---
sudo cp /etc/fstab /etc/fstab.backup.$(date +%Y%m%d_%H%M%S)
echo "UUID=${DISK_UUID}  ${MOUNT_DIR}  ext4  defaults,nofail  0  2" | sudo tee -a /etc/fstab

sudo mount -a && echo "fstab validation passed — all entries mounted successfully."
df -h ${MOUNT_DIR}

LVM — Dynamic Storage That Grows With Your Application

Here's the problem with raw partitions: they're static. You create a 50GB partition for your database, the database grows to 48GB, and now you're racing against time. Your only options are to resize the partition (risky, requires unmounting on most filesystems) or provision a new disk and move data. Neither is fun at 2 AM.

LVM — Logical Volume Manager — solves this by adding an abstraction layer between physical disks and the filesystems sitting on them. Instead of your filesystem sitting directly on /dev/sdb1, it sits on a logical volume that can be expanded by simply adding more physical storage to the underlying pool, called a Volume Group.

The mental model has three layers. Physical Volumes (PVs) are the raw disks or partitions you hand to LVM. A Volume Group (VG) is the pool — LVM combines all your PVs into one big storage bucket. Logical Volumes (LVs) are carved out of that pool and behave like normal partitions from the filesystem's perspective. The magic is that you can extend an LV while it's live and mounted, without unmounting or stopping the application.

This is why nearly every production Linux server uses LVM for everything except /boot. It's not complexity for its own sake — it's the ability to respond to storage demands without downtime.

#!/bin/bash
# lvm_setup_and_extend.sh
# Demonstrates: creating an LVM stack from scratch AND extending a full volume live.
# Scenario: web app data volume (/dev/sdc) is full. We add a new disk (/dev/sdd)
# and expand the logical volume online — zero downtime.

# ============================================================
# PART 1: BUILD AN LVM STACK ON A FRESH DISK
# ============================================================

NEW_DISK="/dev/sdc"
VOLUME_GROUP="webdata_vg"
LOGICAL_VOLUME="webapp_lv"
LV_SIZE="30G"
MOUNT_POINT="/var/www/appdata"

echo "=== STEP 1: Create Physical Volume ==="
sudo pvcreate ${NEW_DISK}
sudo pvdisplay ${NEW_DISK}

echo "=== STEP 2: Create Volume Group ==="
sudo vgcreate ${VOLUME_GROUP} ${NEW_DISK}
sudo vgdisplay ${VOLUME_GROUP}

echo "=== STEP 3: Create Logical Volume ==="
sudo lvcreate -L ${LV_SIZE} -n ${LOGICAL_VOLUME} ${VOLUME_GROUP}
echo "LV device path: /dev/mapper/${VOLUME_GROUP}-${LOGICAL_VOLUME}"

echo "=== STEP 4: Format and Mount the Logical Volume ==="
sudo mkfs.ext4 -L webapp-data /dev/mapper/${VOLUME_GROUP}-${LOGICAL_VOLUME}
sudo mkdir -p ${MOUNT_POINT}
sudo mount /dev/mapper/${VOLUME_GROUP}-${LOGICAL_VOLUME} ${MOUNT_POINT}

DEVICE_PATH="/dev/mapper/${VOLUME_GROUP}-${LOGICAL_VOLUME}"
echo "${DEVICE_PATH}  ${MOUNT_POINT}  ext4  defaults,nofail  0  2" | sudo tee -a /etc/fstab
df -h ${MOUNT_POINT}

# ============================================================
# PART 2: EXTENDING THE VOLUME ONLINE (ZERO DOWNTIME)
# ============================================================

EXTRA_DISK="/dev/sdd"
EXTEND_BY="+50G"

echo "=== EXTEND: Add new disk to the VG pool ==="
sudo pvcreate ${EXTRA_DISK}
sudo vgextend ${VOLUME_GROUP} ${EXTRA_DISK}
sudo vgs ${VOLUME_GROUP}

echo "=== EXTEND: Grow the logical volume ==="
sudo lvextend -L ${EXTEND_BY} -r /dev/mapper/${VOLUME_GROUP}-${LOGICAL_VOLUME}
df -h ${MOUNT_POINT}

Monitoring, Troubleshooting and the /etc/fstab Deep Dive

Understanding how to provision storage is half the job. The other half is knowing when something's going wrong before it takes down your application, and being able to diagnose it fast.

The biggest production risk is a full disk — but the sneaky version is inodes running out before disk space does. Every file on an ext4 filesystem consumes one inode. A directory full of millions of tiny temp files (log shards, session files, cache chunks) can exhaust inodes while df -h shows 40% free space. The symptom is 'No space left on device' errors even though the disk looks fine. df -i reveals the truth.

For performance visibility, iostat from the sysstat package shows read/write throughput and I/O wait per device. High iowait on a specific device tells you whether your application is CPU-bound or storage-bound. iotop shows which processes are doing the most I/O right now — invaluable for finding a runaway process.

For /etc/fstab specifically: the six fields matter. The 'dump' field (5th, almost always 0) controls backup utilities. The 'pass' field (6th) controls fsck order — root should be 1, everything else 2 or 0 to skip. A wrong pass value on a network filesystem causes boot hangs because fsck tries to check an NFS share that isn't available yet.

#!/bin/bash
# storage_monitoring_and_diagnostics.sh
# Production-grade monitoring and diagnostics for Linux storage.
# Covers: disk usage, inode exhaustion, I/O performance, fstab validation.

echo "============================================"
echo " STORAGE HEALTH DASHBOARD"
echo "============================================"

echo ""
echo "[1] DISK SPACE USAGE (human-readable)"
df -hT --exclude-type=tmpfs --exclude-type=devtmpfs

echo ""
echo "[2] INODE USAGE — check this if you see 'No space left on device' with free space"
df -i --exclude-type=tmpfs --exclude-type=devtmpfs

echo ""
echo "[3] TOP 10 LARGEST DIRECTORIES under /var (common culprit for space issues)"
sudo du -sh /var/*/  2>/dev/null | sort -rh | head -10

echo ""
echo "[4] DISK I/O STATISTICS (3-second sample)"
sudo iostat -xd 1 3 2>/dev/null || echo "Install sysstat: sudo apt install sysstat"

echo ""
echo "[5] TOP I/O PROCESSES (requires iotop)"
sudo iotop -b -n 1 -o 2>/dev/null || echo "Install iotop: sudo apt install iotop"

echo ""
echo "[6] FSTAB VALIDATION"
grep -v '^#' /etc/fstab | grep -v '^$' | column -t
echo "Testing fstab by running: mount -a"
sudo mount -a 2>&1 && echo "✓ fstab OK" || echo "✗ fstab ERROR — fix before rebooting!"

echo ""
echo "[7] DISK HEALTH CHECK (smartctl)"
sudo smartctl -H /dev/sda 2>/dev/null || echo "Install smartmontools"

echo ""
echo "[8] CAPACITY ALERTS (>80% used)"
df -h --output=source,pcent,target | awk 'NR>1 && $2+0 > 80 { print "ALERT: " $1 " at " $2 " capacity — mount: " $3 }' || echo "All filesystems under 80%"

LVM Snapshots: Consistent Backups Without Downtime

LVM snapshots let you take a point-in-time copy of a logical volume without unmounting it. They're not backups themselves — they're a consistent image you can then back up. Snapshots use copy-on-write: the original volume continues to be used normally, and the snapshot only stores the original data as it changes. This means snapshots are space-efficient initially, but they grow as writes occur.

The classic use case is database backup. You take a snapshot of the LV containing your MySQL data directory, mount the snapshot somewhere else, and run mysqldump or copy files from the snapshot. The production database keeps running with minimal impact.

Critical: snapshots consume space in the same Volume Group. If the original volume changes too much (writes happen), the snapshot fills up and becomes invalid. You must allocate enough snapshot size or keep the snapshot duration short. A full snapshot is read-only until you extend or remove it.

Another pattern: create a snapshot before a risky operation (e.g., filesystem resize, partition table change). If something goes wrong, you can revert by copying data back from the snapshot.

#!/bin/bash
# lvm_snapshot_and_restore.sh
# Demonstrates: creating an LVM snapshot, mounting it for backup, and removing it.
# Also shows how to restore data from a snapshot (in case of accidental deletion).

TARGET_LV="/dev/webdata_vg/webapp_lv"   # The LV we want to snapshot
SNAPSHOT_NAME="webapp_lv_snap"           # Name for the snapshot
SNAPSHOT_SIZE="5G"                       # Allocate enough to cover expected writes during snapshot lifetime
MOUNT_DIR_BACKUP="/mnt/snapshot_backup"  # Temporary mount for snapshot

echo "=== STEP 1: Create a read-write snapshot of ${TARGET_LV} ==="
sudo lvcreate -s -L ${SNAPSHOT_SIZE} -n ${SNAPSHOT_NAME} ${TARGET_LV}

# Verify the snapshot device exists
sudo lvs | grep ${SNAPSHOT_NAME}

echo "=== STEP 2: Mount the snapshot for backup ==="
sudo mkdir -p ${MOUNT_DIR_BACKUP}
sudo mount /dev/webdata_vg/${SNAPSHOT_NAME} ${MOUNT_DIR_BACKUP}

# Now you can tar, rsync, or run mysqldump from the snapshot mount point.
# The production volume is untouched.
echo "Backing up data from snapshot..."
sudo tar -czf /backups/webapp_$(date +%Y%m%d_%H%M%S).tar.gz -C ${MOUNT_DIR_BACKUP} .

echo "=== STEP 3: Unmount and remove the snapshot ==="
sudo umount ${MOUNT_DIR_BACKUP}
sudo lvremove -f /dev/webdata_vg/${SNAPSHOT_NAME}

echo "Snapshot removed. Backup complete."