OS Interview Questions - Oversized Heap Triggers Swap Storm

Operating system questions are the great equaliser in technical interviews. Whether you're going for a backend role, a systems position, or a cloud engineering job, interviewers reach for OS concepts because they reveal whether you actually understand what happens beneath your code — or whether you've just been writing for loops and calling it engineering. A candidate who understands why a context switch is expensive will write fundamentally different (and better) concurrent code than one who doesn't.

The OS bridges the gap between raw hardware and the applications we write every day. It solves an otherwise impossible coordination problem: dozens of programs all want the CPU, all want memory, all want to read files simultaneously — and the OS makes that work without them knowing about each other. Without it, every application would need to implement its own hardware drivers, scheduling logic, and memory allocation — chaos.

By the end of this article you'll be able to answer the most commonly asked OS interview questions with confidence and depth. You'll understand not just what processes, threads, scheduling, deadlocks, and virtual memory are, but why they were designed that way — which is what separates a good answer from a great one in any technical interview.

Process vs. Thread: The Unit of Execution

One of the most frequent senior-level questions is the architectural difference between a Process and a Thread. A Process is an independent program in execution with its own dedicated memory space (Stack, Heap, Data). A Thread is the smallest unit of execution within a process; all threads of a single process share the same Heap and Code segment but have their own separate Stacks.

From an interviewer's perspective, the 'aha!' moment comes when you discuss Context Switching overhead. Switching between processes is expensive because the OS must flush CPU caches and reload memory maps (TLB). Switching between threads is 'cheaper' but introduces the risk of race conditions, requiring careful synchronization using Mutexes or Semaphores.

package io.thecodeforge.os;

import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

/**
 * TheCodeForge — Demonstrating Thread vs Process mental model.
 * Threads share the same 'Heap' (the counter in this example).
 */
public class ConcurrencyDemo {
    private static int sharedCounter = 0;

    public static void main(String[] args) {
        // Using a Thread Pool to manage units of execution
        try (ExecutorService executor = Executors.newFixedThreadPool(2)) {
            for (int i = 0; i < 1000; i++) {
                executor.submit(() -> {
                    // Critical Section: Without synchronization, this is a Race Condition
                    synchronized (ConcurrencyDemo.class) {
                        sharedCounter++;
                    }
                });
            }
        }
        System.out.println("Final Shared Counter: " + sharedCounter);
    }
}

CPU Scheduling: How the OS Decides Who Runs Next

The CPU scheduler decides which process in the ready queue gets the CPU. Senior interview questions go beyond naming algorithms — they expect you to talk about trade-offs: throughput vs response time, fairness vs efficiency.

Round Robin (quantum = 10-100ms) is the most common time-sharing scheduler. It's fair but can have high switching overhead if quantum is too small. Completely Fair Scheduler (CFS) in Linux uses a red-black tree and targets a weighted fair share based on 'nice' values. CF Schedules deadlines: SCHED_DEADLINE (EDF) for real-time tasks.

A key production insight: Batch jobs can starve interactive tasks if the scheduler isn't tuned. Setting kernel.sched_latency_ns and sched_min_granularity_ns can reduce tail latency in mixed workloads.

# TheCodeForge — View and tune Linux CPU scheduler parameters

# Check current scheduler metrics (for a specific PID's task group)
cat /proc/<pid>/sched

# Check overall scheduler statistics
cat /proc/schedstat

# Tune CFS latency (default 6ms, can reduce for low-latency apps)
echo 3000000 > /proc/sys/kernel/sched_latency_ns   # 3ms
echo 750000 > /proc/sys/kernel/sched_min_granularity_ns

# Set scheduler policy for real-time processes (e.g., Java with -XX:+UseCriticalCMSThreads)
chrt -f -p 99 <pid>   # SCHED_FIFO priority 99
chrt -r -p 50 <pid>   # SCHED_RR priority 50

Deadlock: The Four Conditions and How to Break Them

A deadlock happens when two or more processes are each waiting for a resource that the other holds. The Coffman conditions must all hold simultaneously: Mutual Exclusion, Hold and Wait, No Preemption, Circular Wait. In interviews, you need to explain each and then discuss prevention, avoidance, detection, and recovery.

Prevention: Break one condition. For example, allow preemption (force a process to release resource) or require all resources upfront (Hold and Wait broken). Avoidance: Use Banker's Algorithm or resource allocation graph analysis. Detection: Build a wait-for graph and look for cycles. Recovery: Kill one process or preempt resources.

Production example: A Java application using two database connections with mixed ordering caused a deadlock that brought down a payment service. The fix was to always acquire connections in a fixed global order.

package io.thecodeforge.os;

import java.util.concurrent.locks.Lock;
import java.util.concurrent.locks.ReentrantLock;

/**
 * TheCodeForge — Demonstrates classic deadlock and the fixed-order fix.
 */
public class DeadlockDemo {
    private static final Lock lockA = new ReentrantLock();
    private static final Lock lockB = new ReentrantLock();

    public static void main(String[] args) {
        Thread t1 = new Thread(() -> deadlockSequence1());
        Thread t2 = new Thread(() -> deadlockSequence2());
        t1.start(); t2.start();
    }

    // This order causes deadlock: lockA then lockB in both? No, here it's swapped.
    static void deadlockSequence1() {
        lockA.lock();
        try { Thread.sleep(50); } catch (InterruptedException e) {}
        lockB.lock();
        try { System.out.println("Thread1 got both locks"); }
        finally { lockB.unlock(); lockA.unlock(); }
    }

    static void deadlockSequence2() {
        lockB.lock();
        try { Thread.sleep(50); } catch (InterruptedException e) {}
        lockA.lock();
        try { System.out.println("Thread2 got both locks"); }
        finally { lockA.unlock(); lockB.unlock(); }
    }
}

Memory Management: Paging and Virtual Memory

Why doesn't your app crash the moment you run out of physical RAM? The answer is Virtual Memory. The OS gives every process the illusion that it has a large, contiguous block of memory. In reality, this memory is broken into fixed-size 'Pages'. The OS maps these Virtual Pages to physical 'Frames' in RAM using a Page Table.

When a program tries to access a page that isn't currently in RAM, a Page Fault occurs. The OS then fetches that page from the Disk (Swap space). Senior engineers are expected to know that frequent page faults lead to Thrashing—where the system spends more time swapping pages than actually executing code.

# TheCodeForge — Production OS Diagnostics

# 1. Check system virtual memory statistics (Look for 'si' and 'so' - swap in/out)
vmstat 1 5

# 2. View process-specific memory mappings (Stack, Heap, Libraries)
# Replace [PID] with your Java application's process ID
cat /proc/[PID]/maps | head -n 10

# 3. Check for OOM (Out Of Memory) kills in system logs
dmesg | grep -i "oom-kill"

The Cost of Context Switching and How to Measure It

A context switch is the OS's act of saving state of one process/thread and loading another. It's not just CPU registers — the TLB must be flushed (for processes), CPU caches warm up again, and the kernel scheduler runs. A direct context switch (process) can cost 1-10 microseconds, but the indirect costs (cache misses) can add 100s of microseconds to subsequent instructions.

In production, high context switching often means your thread pool is too large. A typical Java web server with sync I/O and 200 threads on 8 cores will spend more time switching than executing. The fix: tune thread pool size to number of cores for CPU-bound tasks; for I/O-bound use threads = cores / (1 - blocking coefficient).

Tools: vmstat -w 1 shows context switches per second (cs column). pidstat -w shows per-process voluntary vs involuntary switches. perf stat -e context-switches gives precise counts.

# TheCodeForge — Measure context switch overhead

# 1. Watch global context switch rate
vmstat -w 1 | awk '{print $14}' # cs column

# 2. Per-process context switches (voluntary and involuntary)
pidstat -w -I -p <PID> 1 5

# 3. Measure cost of single context switch (approx)
# Use a simple benchmark with taskset and strace:
taskset -c 0 perf stat -e context-switches,cpu-migrations ./your_app