Linux Kernel Architecture Deep-Dive: From Boot to System Calls

Every time you run a command, open a file, or send a network request, the Linux kernel is orchestrating the underlying operations. The kernel is the core of the operating system, managing hardware resources and providing essential services to user-space programs. Understanding its architecture is crucial for system programmers, DevOps engineers, and anyone building performance-critical applications.

Linux follows a monolithic kernel design, meaning all core services (scheduling, memory management, file systems, networking) run in kernel space with full hardware access. However, it supports loadable kernel modules (LKMs) to extend functionality without recompiling. This design offers high performance but requires careful coding to avoid crashes.

In this deep-dive, we'll explore the major subsystems: the process scheduler, memory manager, virtual file system (VFS), and system call interface. We'll also look at real-world production incidents and debugging techniques. By the end, you'll have a solid mental model of how the kernel operates and how to troubleshoot common issues.

1. Overview of Linux Kernel Architecture

The Linux kernel is a monolithic, modular, and preemptive kernel. It runs entirely in kernel space with full access to hardware. Its design is divided into several subsystems:

• Process Scheduler: Manages CPU time among processes.
• Memory Manager: Handles virtual memory, paging, and allocation.
• Virtual File System (VFS): Provides a common interface for different file systems.
• Network Stack: Implements protocols like TCP/IP.
• Inter-Process Communication (IPC): Pipes, signals, shared memory, etc.
• Device Drivers: Interface with hardware.

Unlike microkernels, monolithic kernels have all subsystems in one address space, which reduces context-switch overhead but increases complexity. Linux mitigates this with loadable kernel modules (LKMs) that can be inserted and removed at runtime.

uname -a
cat /proc/version
lsmod | head -10

2. The Process Scheduler: CFS and Task States

The Linux process scheduler decides which process runs next. Since kernel 2.6.23, the default scheduler is the Completely Fair Scheduler (CFS). CFS aims to give each process a fair share of CPU time by maintaining a red-black tree of tasks sorted by virtual runtime (vruntime).

CFS selects the task with the smallest vruntime. When a task runs, its vruntime increases. The scheduler periodically checks and may preempt the current task if another has a smaller vruntime.

# View scheduling policy and priority of a process
ps -eo pid,comm,policy,pri,nice --sort=-pid | head -10
# Show CFS statistics per CPU
cat /proc/sched_debug | grep -A5 'cpu#'
# Change priority of a running process
renice -n 10 -p 1234

3. Memory Management: Virtual Memory and Page Tables

The Linux memory manager handles virtual memory, paging, and allocation. Each process has its own virtual address space, divided into pages (typically 4KB). The kernel maintains page tables to map virtual addresses to physical frames.

The kernel uses demand paging: pages are loaded only when accessed. Swapping moves pages to disk when memory is low.

# View memory usage
free -h
# Show virtual memory areas of a process
cat /proc/self/maps | head -10
# Page fault statistics
vmstat -s | grep -i page

4. Virtual File System (VFS) and System Calls

The Virtual File System (VFS) provides a common interface for different file systems (ext4, XFS, NFS, etc.). It defines generic operations like open, read, write, and close. Each file system implements these operations via a file_operations structure.

System calls are the entry points from user space to kernel. For example, read() triggers sys_read, which goes through VFS to the specific file system.

System call numbers are architecture-specific. On x86_64, sys_read is 0.

# Trace system calls for a command
strace -e trace=open,read,write cat /etc/passwd 2>&1 | head -10
# List all system calls on x86_64
ausyscall --dump | head -10

5. Kernel Modules and Device Drivers

Loadable kernel modules (LKMs) allow extending the kernel without recompiling. Modules are .ko files that can be inserted with insmod or modprobe. They can implement device drivers, file systems, or system calls.

Modules can export symbols for other modules to use. They run in kernel space and have full access to hardware.

Device drivers are a common type of module. Linux classifies devices into: - Character devices: Stream-oriented (e.g., serial ports). - Block devices: Random access (e.g., disks). - Network devices: Packet-oriented.

Writing a simple character driver involves implementing open, read, write, and release operations.

# List loaded modules
lsmod
# Show module information
modinfo ext4
# Load and unload a module
sudo modprobe usb_storage
sudo rmmod usb_storage

6. Inter-Process Communication (IPC) Mechanisms

Linux provides several IPC mechanisms for processes to communicate: - Pipes: Unidirectional byte stream (anonymous or named FIFO). - Signals: Asynchronous notifications (e.g., SIGTERM). - Shared Memory: Fastest IPC, multiple processes access same memory region. - Message Queues: POSIX or System V message passing. - Semaphores: Synchronization primitives. - Sockets: Network communication (also local Unix sockets).

Shared memory is implemented via shmget/shmat. The kernel creates a shared memory segment that processes attach to their address space. Synchronization is needed to avoid race conditions.

Signals are handled by the kernel delivering a signal to a process. The process can catch, ignore, or block signals. SIGKILL and SIGSTOP cannot be caught.

# List IPC resources
ipcs -a
# Create a shared memory segment
ipcmk -M 1024
# Remove a shared memory segment
ipcrm -M 0x00000000
# Send a signal to a process
kill -SIGTERM 1234

7. Booting the Kernel: From BIOS to init

The boot process involves several stages: 1. BIOS/UEFI: Initializes hardware and loads the bootloader. 2. Bootloader (GRUB): Loads the kernel and initramfs into memory. 3. Kernel decompression: The kernel decompresses itself and sets up basic memory management. 4. Architecture setup: Initializes CPU, interrupts, and page tables. 5. Kernel initialization: Calls start_kernel(), which initializes subsystems (scheduler, memory, VFS, etc.). 6. init process: The kernel mounts root filesystem and starts the init process (systemd or init).

The kernel command line (e.g., from GRUB) can pass parameters like 'quiet', 'single', or 'mem=1024M'.

# View kernel boot messages
dmesg | head -20
# Check kernel command line
cat /proc/cmdline
# Show boot time of kernel
systemd-analyze