Memory Management Thrashing — 120GB Working Set on 64GB

Every program you run — a browser, a game, a database — needs memory to breathe. Without a fair, structured way to hand out that memory, one misbehaving app could read your bank app's data, a crashed process could corrupt the entire system, and you'd never be able to run more than one program at a time. Memory management is the silent contract that makes modern computing safe and multi-tasking possible.

The problem it solves is deceptively deep. Physical RAM is finite and shared. Process A shouldn't be able to peek into Process B's address space. The OS needs to allocate memory fast, reclaim it when a process exits, and give each program the illusion that it owns all the memory in the world — even when RAM is nearly full. Without a memory manager, none of that is possible.

By the end of this article you'll understand exactly how the OS partitions memory, why paging replaced older schemes, how virtual memory lets your laptop run 40 browser tabs on 8 GB of RAM, and what questions about memory management reveal in a system-design or OS interview. Let's dig in.

What is Memory Management in OS?

Memory management is the OS subsystem responsible for allocating and deallocating memory to processes, tracking which parts of memory are free or in use, and providing isolation so that one process cannot access another's data. At its core, it solves three problems: sharing of finite physical RAM, protection between processes, and translation from virtual to physical addresses. Every modern OS—Linux, Windows, macOS—implements memory management as part of the kernel, using hardware support from the CPU's MMU (Memory Management Unit). Without it, a bug in a browser could corrupt a password manager in memory, and multiprogramming would be impossible.

#include <stdio.h>
#include <unistd.h>

int main() {
    int var = 42;
    printf("Virtual address of var: %p\n", (void*)&var);
    printf("Process PID: %d\n", getpid());
    printf("Check /proc/%d/maps for virtual memory layout\n", getpid());
    // Run: cat /proc/PID/maps to see the address space
    return 0;
}

Memory Allocation Strategies: Contiguous vs Non-Contiguous

Early operating systems used contiguous allocation: a process gets a single block of physical memory. That led to fragmentation and overcommit problems. Modern OSes use non-contiguous allocation via paging. But within a process, memory allocation requests (malloc) are served by the heap manager, which uses a mix of contiguity and segmentation. The two main strategies are:

1. Contiguous (Fixed Partitioning): Each process gets a fixed-size block; simple but wastes memory (external fragmentation). Not used in general-purpose OSes.
2. Non-Contiguous (Paging): Physical memory is split into fixed-size frames; processes get virtual pages that can map to any frame. This eliminates external fragmentation and enables virtual memory.

Additionally, Segmentation allows variable-sized logical chunks (code, data, stack) but suffers from external fragmentation unless combined with paging. Most modern systems (Linux, Windows) use paged segmentation where segments are further broken into pages.

#include <stdio.h>
#include <stdlib.h>

// Simulate a page table for a process with 4 virtual pages
#define NUM_PAGES 4

struct page_table_entry {
    int valid;      // 1 if page is in physical memory
    int frame_num;  // physical frame number (if valid)
    int dirty;      // modified flag for swapping
    int referenced; // for LRU approximation
};

int main() {
    struct page_table_entry pt[NUM_PAGES];
    pt[0] = (struct page_table_entry){1, 10, 0, 0};
    pt[1] = (struct page_table_entry){1, 5, 1, 0};
    pt[2] = (struct page_table_entry){0, 0, 0, 0}; // page fault will occur
    pt[3] = (struct page_table_entry){1, 8, 0, 0};

    int virtual_page = 2;
    if (pt[virtual_page].valid) {
        int physical_addr = pt[virtual_page].frame_num * 4096; // 4KB pages
        printf("Physical address for vaddr %d: 0x%x\n", virtual_page, physical_addr);
    } else {
        printf("Page fault: must load from swap\n");
    }
    return 0;
}

Paging: The Mechanism Behind Virtual Memory

Paging divides virtual memory into fixed-size blocks called pages (typically 4KB on x86_64) and physical memory into frames of the same size. Each process has a page table that maps virtual page numbers to physical frame numbers. The MMU uses this page table to translate every memory access from virtual to physical address. When a process accesses a page not currently in physical memory, a page fault occurs, and the OS loads the page from disk (swap) or from the file system (demand paging).

Page tables themselves are hierarchical (e.g., 4-level page tables on x86_64) to avoid needing a flat table with billions of entries. The Translation Lookaside Buffer (TLB) caches recent translations to speed up address translation. On a context switch, the TLB must be flushed, which is why repeated context switching hurts performance.

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <sys/resource.h>

int main() {
    struct rusage usage_before, usage_after;
    getrusage(RUSAGE_SELF, &usage_before);

    // Cause a page fault by accessing a page that hasn't been touched
    int *big = malloc(1024 * 1024 * 100); // allocate 100 MB
    for (int i = 0; i < 100 * 1024 * 1024 / sizeof(int); i += 4096 / sizeof(int)) {
        big[i] = i; // touch every page boundary
    }

    getrusage(RUSAGE_SELF, &usage_after);
    long page_faults = usage_after.ru_majflt - usage_before.ru_majflt;
    printf("Major page faults: %ld\n", page_faults);
    free(big);
    return 0;
}

Virtual Memory: The Illusion of Infinite Memory

Virtual memory extends the concept of paging: each process gets a full virtual address space (e.g., 2^48 on x86_64), but only the parts actively needed are backed by physical memory. The rest sits on disk (swap). When a process accesses a virtual address that is not in RAM, the OS loads the corresponding page from disk into a freed frame—this is demand paging. If no free frames exist, the OS evicts a page to disk using a replacement algorithm (LRU, Clock, etc.). Virtual memory enables: - Running programs larger than physical RAM. - Sharing libraries and memory-mapped files. - Copy-on-write (COW) forking.

The key components: page table, swap space (disk area), page replacement algorithm, and the page fault handler.

from collections import OrderedDict

def simulate_lru(pages, num_frames):
    frames = OrderedDict()
    page_faults = 0
    for page in pages:
        if page in frames:
            frames.move_to_end(page)  # mark as recently used
        else:
            page_faults += 1
            if len(frames) >= num_frames:
                frames.popitem(last=False)  # remove LRU
            frames[page] = True
    return page_faults

# Access pattern: start of loop, then large working set
accesses = [1, 2, 3, 4, 1, 2, 5, 1, 2, 3, 4, 5]
print(simulate_lru(accesses, 3))  # Output: 8 (most page faults)

Page Replacement Algorithms: Which Pages Get Evicted?

When physical memory is full and a page fault occurs, the OS must evict a victim page to disk. The replacement algorithm determines which page to remove. The goal is to minimize future page faults by evicting pages unlikely to be used soon.

LRU (Least Recently Used): Evict the page not accessed for the longest time. Requires hardware support (reference bits) or software approximation (e.g., Clock algorithm).

Clock (Second Chance): Use a circular list with a reference bit. Sweep through, clearing bits; if bit is already clear, evict. Efficient approximation of LRU.

Working Set Model: Estimate the set of pages a process is actively using; only keep that set in RAM. Prevents thrashing by adjusting degree of multiprogramming.

Other algorithms: FIFO (simple but suffers from Belady's anomaly), Optimal (unimplementable, used as comparison).

#include <stdio.h>
#include <stdlib.h>

#define NUM_FRAMES 4

struct frame {
    int page;
    int referenced; // 0 or 1
};

int main() {
    struct frame frames[NUM_FRAMES] = {{1,0},{2,0},{3,0},{4,0}};
    int hand = 0; // clock hand
    int new_page = 5;

    // Clock algorithm: find a victim
    while (1) {
        if (frames[hand].referenced == 0) {
            printf("Evicting page %d from frame %d\n", frames[hand].page, hand);
            frames[hand].page = new_page;
            frames[hand].referenced = 0;
            break;
        } else {
            frames[hand].referenced = 0; // give second chance
            hand = (hand + 1) % NUM_FRAMES;
        }
    }
    return 0;
}

Fragmentation: The Silent Performance Killer

Fragmentation is what happens when the OS's memory allocation decisions come back to bite you. Two flavors ruin your day. Internal fragmentation wastes memory inside an allocated block. Give a 10KB process a fixed 16KB partition? You just blew 6KB on nothing. External fragmentation is worse: free memory exists, but it's scattered into tiny, unusable holes. After enough allocate-free cycles, you get a Swiss cheese heap. No single free block can satisfy a large request. Your system starts thrashing, swapping pages like a gambler chasing losses. The fix? Non-contiguous schemes like paging. By breaking memory into fixed-size frames and processes into pages, the OS sidesteps external fragmentation entirely. Internal fragmentation becomes minimal — at most one partial page per process. If you're still seeing allocation failures on a box with free memory, check your fragmentation. It's the first thing I grep for in a production outage.

// io.thecodeforge
#include <stdio.h>
#include <stdlib.h>

void check_contiguous_fragmentation(size_t *free_blocks, int count, size_t needed) {
    size_t largest = 0;
    for (int i = 0; i < count; i++) {
        if (free_blocks[i] >= needed) {
            printf("Found viable block: %zu bytes\n", free_blocks[i]);
            return;
        }
        if (free_blocks[i] > largest) largest = free_blocks[i];
    }
    printf("EXTERNAL FRAGMENTATION: Largest free block = %zu, Requested = %zu\n", largest, needed);
    // Output: EXTERNAL FRAGMENTATION: Largest free block = 40, Requested = 64
}

int main() {
    size_t blocks[] = {10, 4, 20, 6};  // total free = 40, but no contiguous 64
    check_contiguous_fragmentation(blocks, 4, 64);
    return 0;
}

Modern Memory Allocation: jemalloc, tcmalloc, mimalloc

Traditional malloc implementations like glibc's ptmalloc2 can suffer from fragmentation and scalability issues under multi-threaded workloads. Modern allocators address these problems with per-thread caches, lock-free data structures, and optimized memory layouts.

jemalloc (used by Facebook, Redis, Firefox) reduces fragmentation by using separate arenas for different thread groups and maintaining size classes with buddy allocation. It excels in multi-threaded environments by minimizing contention.

tcmalloc (Google's allocator) uses thread-local caches and a central heap. It batches small allocations into pages and uses a page-level free list. This reduces lock contention and improves cache locality.

mimalloc (Microsoft) focuses on free list sharding and eager page purging. It uses a compact metadata structure and a novel 'free list of free lists' approach to reduce fragmentation.

Practical Example: Consider a web server handling 1000 concurrent requests, each allocating and freeing small objects. With ptmalloc, lock contention on the central heap can cause thrashing. Switching to jemalloc reduces contention by using per-thread arenas, improving throughput by 30%.

Production Insight: When migrating to a modern allocator, benchmark with realistic workloads. For example, Redis uses jemalloc by default; switching to mimalloc improved latency by 5% in some tests. Always test with your specific allocation patterns.

#include <stdlib.h>
#include <pthread.h>
#include <stdio.h>

#define NUM_THREADS 100
#define ALLOCS_PER_THREAD 100000

void* worker(void* arg) {
    for (int i = 0; i < ALLOCS_PER_THREAD; i++) {
        void* p = malloc(64);
        free(p);
    }
    return NULL;
}

int main() {
    pthread_t threads[NUM_THREADS];
    for (int i = 0; i < NUM_THREADS; i++)
        pthread_create(&threads[i], NULL, worker, NULL);
    for (int i = 0; i < NUM_THREADS; i++)
        pthread_join(threads[i], NULL);
    printf("Done\n");
    return 0;
}

Huge Pages and Transparent Huge Pages in Linux

Huge pages reduce TLB misses by mapping large contiguous memory regions with fewer page table entries. Standard 4KB pages can cause TLB thrashing for workloads with large working sets (e.g., databases, VMs). Linux supports 2MB and 1GB huge pages.

Explicit Huge Pages: Reserved at boot or via /proc/sys/vm/nr_hugepages. Applications use mmap with MAP_HUGETLB or hugetlbfs. This guarantees huge pages but requires manual management.

Transparent Huge Pages (THP): Automatically promotes eligible 4KB pages to 2MB huge pages. Enabled by default on many distributions. However, THP can cause latency spikes due to compaction (memory defragmentation) and increased memory usage.

Practical Example: A Redis instance with 50GB dataset on a 64GB machine. Without huge pages, TLB misses cause 5% CPU overhead. Enabling THP reduces TLB misses by 80%, but compaction pauses increase latency by 2ms. Using explicit huge pages avoids compaction but requires memory reservation.

Production Insight: For latency-sensitive applications, disable THP (echo never > /sys/kernel/mm/transparent_hugepage/enabled) and use explicit huge pages. For throughput-oriented workloads, THP can be beneficial. Monitor /proc/meminfo for HugePages_Total and HugePages_Free.

# Reserve 512 2MB huge pages (1GB total)
echo 512 > /proc/sys/vm/nr_hugepages

# Check reservation
cat /proc/meminfo | grep HugePages

# Mount hugetlbfs
mkdir -p /mnt/huge
mount -t hugetlbfs hugetlbfs /mnt/huge

# Use in C program
// mmap with MAP_HUGETLB flag
void* ptr = mmap(NULL, 2*1024*1024, PROT_READ|PROT_WRITE,
                 MAP_PRIVATE|MAP_ANONYMOUS|MAP_HUGETLB, -1, 0);

Memory Overcommit and OOM Killer

Linux overcommits memory by default: it allows processes to allocate more virtual memory than physical RAM + swap. This relies on the fact that applications rarely use all allocated memory. However, when actual memory usage exceeds capacity, the Out-Of-Memory (OOM) Killer terminates processes to free memory.

Overcommit Modes: - 0 (heuristic overcommit): Based on overcommit ratio. - 1 (always overcommit): Never refuse malloc. - 2 (no overcommit): Fail if allocation exceeds commit limit.

OOM Killer Selection: Based on oom_score (badness) which considers memory usage, runtime, and root privileges. The process with the highest score is killed.

Practical Example: A Java application with -Xmx80G on a 64GB machine. With overcommit enabled, the JVM starts successfully. When the application actually uses 70GB and swap is full, the OOM Killer kills a random process (e.g., SSH daemon). To avoid this, set vm.overcommit_memory=2 and vm.overcommit_ratio=50 (50% of RAM+swap).

Production Insight: For critical services, disable overcommit (vm.overcommit_memory=2) and set vm.overcommit_ratio appropriately. Monitor /proc/meminfo for Committed_AS (total committed memory). Use cgroups to limit memory per process and avoid OOM killing unrelated services.

# Disable overcommit (strict mode)
echo 2 > /proc/sys/vm/overcommit_memory

# Set overcommit ratio to 50% of RAM+swap
echo 50 > /proc/sys/vm/overcommit_ratio

# Check current committed memory
cat /proc/meminfo | grep Committed_AS

# Adjust OOM score for a process (lower = less likely to be killed)
echo -1000 > /proc/1234/oom_score_adj