Linux Process Management — Unkillable D State from NFS Hang

Every command you run, every web server you start, every cron job that fires at midnight — all of it becomes a Linux process. Understanding how those processes live, communicate and die isn't optional knowledge for a DevOps engineer or backend developer; it's the difference between confidently diagnosing a runaway process at 2 AM and blindly rebooting a production server and hoping for the best.

The problem most developers hit is that they learn 'ps aux | grep something' and 'kill -9' and think that's process management. It isn't. That's like learning to use a fire extinguisher but not knowing what causes fires. Real process management means understanding process states, parent-child relationships, signals, job control, and how the kernel schedules work — so you can make deliberate decisions instead of panicked ones.

By the end of this article you'll be able to inspect any running process and understand what it's doing, send the right signal for the right situation (spoiler: kill -9 is almost never the right answer), manage foreground and background jobs like a pro, and build the mental model that makes every 'why is my server slow?' investigation start from a place of clarity.

How Linux Processes Are Born — PIDs, PPIDs and the Process Tree

Every process in Linux gets a Process ID (PID) — a unique integer the kernel assigns at birth. But processes don't appear from nowhere. Almost every process is spawned by another process, its parent, which holds a Parent Process ID (PPID). This parent-child relationship forms a tree, and the root of that entire tree is PID 1 — the init system (systemd on modern distros).

Why does this matter? Because when a parent process dies before its child, the child becomes an 'orphan' and gets re-parented to PID 1. When a child dies but the parent hasn't called wait() to collect its exit status, the child becomes a 'zombie' — it occupies a PID slot and a row in the process table while holding no real resources. A handful of zombies is harmless. Thousands mean something in your code is seriously wrong.

The fork-exec model is how new processes are created. A parent calls fork(), which clones itself into a child process. The child then calls exec() to replace its memory with a new program. This is why your shell is the parent of almost every command you run — and why killing your terminal kills the processes running inside it.

Run pstree to see the entire family tree live. It's one of the most clarifying commands a Linux learner can run.

#!/usr/bin/env bash
# process_tree_inspection.sh
# Goal: Understand where a process comes from and who owns it

# --- Step 1: Find the PID of the current shell ---
current_shell_pid=$$   # $$ is a special variable holding the current process's PID
echo "Current shell PID: $current_shell_pid"

# --- Step 2: Start a background sleep to give us something to inspect ---
sleep 300 &            # The & sends the process to the background immediately
sleep_pid=$!           # $! captures the PID of the last background command
echo "Background sleep PID: $sleep_pid"

# --- Step 3: Inspect the process in detail using ps ---
# -o lets us choose exactly which columns to display
# pid=process id, ppid=parent process id, stat=state, cmd=full command
echo ""
echo "--- Detailed view of our sleep process ---"
ps -o pid,ppid,stat,user,cmd -p "$sleep_pid"

# --- Step 4: Show how this process fits in the full tree ---
echo ""
echo "--- Process tree rooted at our shell ---"
pstree -p "$current_shell_pid"   # -p shows PIDs next to each process name

# --- Step 5: Clean up — kill our background sleep gracefully ---
kill "$sleep_pid"      # Sends SIGTERM (signal 15) by default — polite shutdown request
echo ""
echo "Sent SIGTERM to sleep process $sleep_pid. It should be gone now."

# --- Step 6: Confirm it's gone ---
sleep 0.2              # Give the kernel a moment to clean up
if ! kill -0 "$sleep_pid" 2>/dev/null; then
  # kill -0 doesn't actually kill — it just checks if the process exists
  echo "Confirmed: PID $sleep_pid no longer exists."
fi

Reading Process State — What ps and top Are Actually Telling You

Developers glance at ps output and look for a name. Senior engineers look at the STAT column first. That single letter (or two) tells you exactly what the kernel is doing with that process right now, and it's the fastest way to diagnose a sick system.

The core states are: R (Running or runnable — actively using CPU or waiting for a CPU slot), S (Interruptible Sleep — waiting for I/O or an event, will wake up when signalled), D (Uninterruptible Sleep — waiting on I/O it cannot be interrupted from, typically disk or NFS), Z (Zombie — dead but parent hasn't collected exit status), and T (Stopped — paused by a signal like SIGSTOP or by a debugger).

The D state is the one that causes real pain. A process in D state cannot be killed — not even with kill -9. It's waiting on the kernel for something and is completely outside the kill path until that kernel operation finishes or times out. If you see dozens of processes in D state, your storage layer is almost certainly the problem: a hung NFS mount, a failing disk, or an overloaded I/O scheduler.

top and htop give you the same state information but in real time, so you can watch a process oscillate between R and S as it processes requests — that's healthy. A process pinned in R consuming 100% CPU for minutes is not.

#!/usr/bin/env bash
# process_state_diagnosis.sh
# Goal: Show how to read process states and identify problematic ones

# --- Snapshot of all processes with state info ---
# a = all users, u = user-oriented format, x = include processes without a terminal
echo "=== Full process snapshot (top 20 by CPU) ==="
ps aux --sort=-%cpu | head -20
# Output columns: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

echo ""
echo "=== Processes currently in Uninterruptible Sleep (D state) ==="
# These are the dangerous ones — they can't be killed and indicate I/O problems
ps aux | awk '$8 ~ /^D/ { print $0 }'
# awk checks column 8 (STAT) for a value starting with D

echo ""
echo "=== Zombie processes on this system ==="
# Zombies start with Z in the STAT column
zombie_count=$(ps aux | awk '$8 ~ /^Z/ { count++ } END { print count+0 }')
echo "Zombie count: $zombie_count"
if [ "$zombie_count" -gt 0 ]; then
  echo "Zombies found — listing with parent PIDs:"
  ps aux | awk '$8 ~ /^Z/ { print $0 }'
  echo ""
  echo "To fix zombies: identify the parent (PPID) and restart it."
  echo "The parent is responsible for calling wait() to reap its children."
fi

echo ""
echo "=== Top 5 memory consumers ==="
ps aux --sort=-%mem | awk 'NR==1 || NR<=6 { print $0 }'
# NR==1 preserves the header row, NR<=6 gives us 5 data rows

Signals — The Right Way to Talk to a Running Process

A signal is a small integer the kernel delivers to a process as a notification or instruction. Most developers only know two: kill -9 and 'the other one'. That ignorance causes real production problems — from data corruption when processes don't get to flush their write buffers, to configuration changes never taking effect because an engineer restarted instead of reloaded.

The key signals every DevOps engineer must know: SIGTERM (15) is a polite shutdown request — the process can catch this, finish what it's doing, close files, and exit cleanly. This is the default signal for kill and the one you should try first. SIGKILL (9) is unconditional termination by the kernel — the process gets no say, no cleanup. Use it only when SIGTERM has failed after a reasonable wait. SIGHUP (1) means 'hang up' and historically disconnected modems, but modern daemons like nginx and sshd re-read their config files when they receive SIGHUP — no restart, no downtime. SIGSTOP (19) and SIGCONT (18) pause and resume a process, identical to what Ctrl+Z and fg do from your terminal. SIGUSR1 and SIGUSR2 are user-defined signals that applications can use for custom behaviour — some log rotation tools use these.

The kill command is misnamed — it sends signals, it doesn't exclusively kill. kill -l shows every signal your system supports.

#!/usr/bin/env bash
# signal_management_demo.sh
# Goal: Demonstrate the right signal for each situation

# --- Part 1: Graceful shutdown vs forced kill ---
# Start a simulated long-running service
sleep 600 &
long_running_pid=$!
echo "Started fake service with PID: $long_running_pid"

# The RIGHT first step — ask politely
echo "Sending SIGTERM (graceful shutdown request)..."
kill -SIGTERM "$long_running_pid"   # Same as: kill -15 $long_running_pid

# Wait up to 5 seconds for graceful shutdown
for wait_seconds in 1 2 3 4 5; do
  sleep 1
  if ! kill -0 "$long_running_pid" 2>/dev/null; then
    echo "Process exited cleanly after ${wait_seconds}s. Good."
    break
  fi
  if [ "$wait_seconds" -eq 5 ]; then
    echo "Process didn't respond to SIGTERM after 5s. Now using SIGKILL."
    kill -SIGKILL "$long_running_pid"   # Only escalate when SIGTERM fails
  fi
done

echo ""

# --- Part 2: SIGHUP for zero-downtime config reload ---
# In production you'd do: kill -SIGHUP $(cat /var/run/nginx.pid)
# Let's simulate it with a script that catches SIGHUP
cat > /tmp/signal_catcher.sh << 'SCRIPT'
#!/usr/bin/env bash
trap 'echo "[PID $$] Caught SIGHUP — reloading config (no restart needed)"' SIGHUP
trap 'echo "[PID $$] Caught SIGTERM — shutting down cleanly"; exit 0' SIGTERM
echo "[PID $$] Service started. Waiting for signals..."
while true; do sleep 1; done
SCRIPT
chmod +x /tmp/signal_catcher.sh

/tmp/signal_catcher.sh &
catcher_pid=$!
sleep 0.5   # Give it a moment to start

echo "--- Simulating config reload with SIGHUP ---"
kill -SIGHUP "$catcher_pid"         # nginx does this — reload config, keep serving traffic
sleep 0.3

echo ""
echo "--- Simulating graceful shutdown with SIGTERM ---"
kill -SIGTERM "$catcher_pid"        # Clean exit
wait "$catcher_pid" 2>/dev/null     # Wait for it to finish before exiting this script

echo ""
echo "--- All signals on this system (for reference) ---"
kill -l   # Print all signal names and numbers

Job Control — Managing Foreground, Background and Suspended Processes

Job control is the shell's built-in mechanism for managing multiple processes from a single terminal session. It's the feature that lets you start a long compile, push it to the background, check your email, bring the compile back, and do all of this without opening a second terminal.

When you press Ctrl+Z, the shell sends SIGTSTP to the foreground process, which pauses it immediately (state changes to T). The process is now a 'stopped job'. bg resumes it in the background (sends SIGCONT). fg brings any background or stopped job back to the foreground. The jobs command lists everything the current shell is managing.

The critical thing most developers miss is that background jobs in a terminal session are tied to that terminal. Close the terminal (or SSH connection drops), and the shell sends SIGHUP to all its jobs, which kills them. This is why nohup and disown exist — nohup makes a process immune to SIGHUP, and disown removes a job from the shell's job table so closing the terminal doesn't affect it.

For anything that needs to truly survive a disconnection, use tmux or screen — they create a persistent session that lives on the server, not inside your SSH connection.

#!/usr/bin/env bash
# job_control_workflow.sh
# Goal: Show the complete job control lifecycle including background survival

# --- Part 1: Basic job management ---
echo "=== Starting three background jobs ==="

# Simulate three different long-running tasks
sleep 120 &   # Pretend this is a database backup
backup_pid=$!
echo "Backup job started — PID: $backup_pid, Job: $!"

sleep 240 &   # Pretend this is a data export
export_pid=$!
echo "Export job started — PID: $export_pid"

sleep 360 &   # Pretend this is a log archive
archive_pid=$!
echo "Archive job started — PID: $archive_pid"

echo ""
echo "=== All current jobs in this shell ==="
jobs -l   # -l includes PIDs alongside job numbers

echo ""
echo "=== Suspending the export job (simulating Ctrl+Z) ==="
kill -SIGTSTP "$export_pid"   # Same signal as pressing Ctrl+Z interactively
sleep 0.2

echo "Job state after SIGTSTP:"
jobs -l   # Export should now show as 'Stopped'

echo ""
echo "=== Resuming export in the background ==="
bg %2    # %2 refers to job number 2 (the export). bg sends SIGCONT
sleep 0.2
jobs -l  # Should be back to Running

echo ""
# --- Part 2: Making a job survive terminal closure ---
echo "=== Running a job that survives SSH disconnection ==="

# nohup redirects stdout/stderr to nohup.out and makes process immune to SIGHUP
nohup sleep 9999 > /tmp/persistent_job.log 2>&1 &
persistent_pid=$!
echo "Persistent job PID: $persistent_pid"

# disown removes it from shell job table — terminal closing won't affect it
disown "$persistent_pid"
echo "Job $persistent_pid disowned — it will survive terminal closure"

# Verify it's no longer in the job table
echo ""
echo "Current jobs (persistent_pid should NOT appear):"
jobs -l

echo ""
echo "But it IS still in the process table:"
ps -p "$persistent_pid" -o pid,stat,cmd

# --- Cleanup ---
kill "$backup_pid" "$export_pid" "$archive_pid" "$persistent_pid" 2>/dev/null
wait 2>/dev/null
echo ""
echo "All jobs cleaned up."

Debugging Running Processes with strace and ltrace

When a process is misbehaving — high CPU, hanging, slow responses — the first question is 'what is it actually doing right now?' strace gives you the answer by intercepting system calls: every open, read, write, connect, poll that the process makes. ltrace does the same for library calls (e.g., malloc, free, gettimeofday).

Common debugging scenarios: A web server that's slow — strace -p <PID> -e trace=network reveals it's stuck on a connect() to a backend that's not responding. A process consuming 100% CPU — strace -c -p <PID> shows the distribution of syscall counts; if you see millions of gettimeofday() calls, your code is polling in a tight loop. A process that's leaking memory — strace -e trace=brk,mmap,munmap -p <PID> shows every heap allocation and deallocation.

strace can also attach to already-running processes, follow child processes (-f), and filter by specific syscalls (-e). Use it sparingly in production because it slows the traced process significantly (often 10-100x slower syscalls). For quick checks, strace -p <PID> -c for a summary, then dive deeper if needed.

ltrace is less common but useful when you suspect a library call is the bottleneck — for example, a process that calls gettimeofday millions of times or does excessive memory allocation.

#!/usr/bin/env bash
# strace_debug_demo.sh
# Goal: Use strace to diagnose process behavior without modifying the process

# Simulate a problematic process: a tight loop calling gettimeofday()
cat > /tmp/busy_loop.py << 'PYTHON'
import time
import sys
while True:
    time.time()  # calls gettimeofday syscall
    if time.time() % 1000 < 0.001:
        print("tick")
PYTHON

python3 /tmp/busy_loop.py &
busy_pid=$!
sleep 0.5  # Let it start

echo "=== strace summary for PID $busy_pid (10 seconds) ==="
strace -p "$busy_pid" -c -S time 2>&1 &
strace_pid=$!
sleep 3
kill "$strace_pid" 2>/dev/null
wait "$strace_pid" 2>/dev/null

echo ""
echo "=== Showing last 10 syscalls for PID $busy_pid ==="
strace -p "$busy_pid" -e trace=write -c -S calls 2>&1 &
strace_pid2=$!
sleep 2
kill "$strace_pid2" 2>/dev/null
wait "$strace_pid2" 2>/dev/null

echo ""
echo "=== Killing the test process ==="
kill "$busy_pid" 2>/dev/null
wait "$busy_pid" 2>/dev/null
echo "Done."