Linux & SRE Interview Questions: Advanced DevOps Guide

Linux and Site Reliability Engineering (SRE) are the backbone of modern DevOps. In an interview, you'll be tested not just on commands but on deep understanding of system internals, troubleshooting methodologies, and reliability principles. This guide covers advanced topics like process management, memory analysis, filesystem debugging, networking, and SRE concepts such as SLIs, SLOs, error budgets, and incident response. We'll walk through real interview questions with solutions, production incidents, and debugging cheat sheets. By the end, you'll be ready to ace any Linux or SRE interview and demonstrate your ability to keep large-scale systems reliable.

1. Linux Process Management

Process management is a core Linux interview topic. You need to understand process states, signals, and how to troubleshoot. Common questions: 'How do you find a process using the most memory?' or 'What is a zombie process and how do you fix it?' Use ps, top, htop, and /proc filesystem. For example, to list top memory consumers: ps aux --sort=-%mem | head. A zombie process is a child that has terminated but its parent hasn't reaped it. You can kill the parent or send SIGCHLD. In an interview, explain the process lifecycle: fork, exec, wait. Also discuss signals like SIGTERM, SIGKILL, and SIGSTOP. Practical tip: Use strace to trace system calls of a process. Example: strace -p 1234 -e trace=network to see network calls.

#!/bin/bash
# Find top CPU consuming processes
ps -eo pid,ppid,cmd,%cpu,%mem --sort=-%cpu | head -10

# Check for zombie processes
ps aux | awk '{if ($8 == "Z") print $0}'

# Trace system calls of a process (replace PID)
strace -p 1234 -e trace=open,read,write 2>&1 | head -20

2. Memory Management and Analysis

Memory issues are common in production. Interviewers ask: 'How do you check memory usage?' or 'What is swap and when is it used?' Use free -m, vmstat, and /proc/meminfo. Understand the difference between virtual and resident memory. A memory leak can be detected by monitoring top over time. Use pmap -x PID to see memory mapping. In an interview, explain the OOM killer: when memory is exhausted, the kernel kills a process based on oom_score. You can adjust oom_score_adj to protect critical services. Example: echo -1000 > /proc/PID/oom_score_adj. Also discuss swap: swap is used when physical memory is full. High swap usage indicates memory pressure. Use swapon --show to see swap usage.

#!/bin/bash
# Check memory usage
echo "=== Memory Usage ==="
free -h

# Check swap usage
echo "=== Swap ==="
swapon --show

# Top memory consuming processes
echo "=== Top Memory Processes ==="
ps aux --sort=-%mem | head -5

# Detailed memory map for a process (replace PID)
# pmap -x 1234 | tail -20

3. Filesystem and Disk Management

Filesystem questions test your knowledge of disk usage, inodes, and mount points. Common: 'How do you find large files?' or 'What is an inode?' Use df -h for disk usage, du -sh * for directory sizes, and ls -i for inode numbers. To find files older than 30 days: find / -type f -mtime +30. Understand filesystem types: ext4, XFS, Btrfs. In an interview, discuss the difference between hard links and soft links. Hard links share the same inode; soft links are pointers. Also discuss lsof to find open files. Example: lsof | grep deleted shows files that are deleted but still held open, causing disk space not to be freed.

#!/bin/bash
# Check disk usage
echo "=== Disk Usage ==="
df -h

# Find large files (>100MB)
echo "=== Large Files ==="
find / -type f -size +100M -exec ls -lh {} \; 2>/dev/null

# Check inode usage
echo "=== Inode Usage ==="
df -i

# Find open deleted files
echo "=== Open Deleted Files ==="
lsof | grep deleted

4. Networking Fundamentals

Networking is critical for SRE. Interviewers ask: 'How do you check listening ports?' or 'How do you troubleshoot connectivity issues?' Use ss -tlnp to see listening TCP ports, netstat (deprecated but still used), and tcpdump for packet capture. Understand the OSI model and TCP/IP stack. Common tools: ping, traceroute, mtr, nslookup, dig. For performance, use iperf to test bandwidth. In an interview, explain how to check for packet loss: netstat -s | grep -i loss. Also discuss socket states: LISTEN, ESTABLISHED, TIME_WAIT. Many TIME_WAIT sockets can indicate connection pooling issues. Example: ss -s shows socket statistics.

#!/bin/bash
# Check listening ports
echo "=== Listening Ports ==="
ss -tlnp

# Check socket statistics
echo "=== Socket Stats ==="
ss -s

# Capture packets on port 80 (run with sudo)
# tcpdump -i eth0 port 80 -c 10

# Test connectivity
echo "=== Ping Test ==="
ping -c 4 google.com

5. Systemd and Service Management

Modern Linux uses systemd. Interviewers ask: 'How do you create a systemd service?' or 'How do you view logs?' Use systemctl to manage services: systemctl start|stop|restart|status. Logs are viewed with journalctl. Example: journalctl -u nginx -f to follow logs. Understand unit files: location /etc/systemd/system/. A simple service file includes [Unit], [Service], [Install] sections. In an interview, explain the difference between Type=simple and Type=forking. Also discuss dependencies: After=network.target. For debugging, use systemd-analyze blame to see boot time.

[Unit]
Description=My Custom Application
After=network.target

[Service]
Type=simple
User=myapp
ExecStart=/usr/local/bin/myapp
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

6. SRE Principles: SLIs, SLOs, and Error Budgets

SRE interviews focus on reliability metrics. SLI (Service Level Indicator) is a measurement, e.g., latency. SLO (Service Level Objective) is a target, e.g., 99.9% of requests under 200ms. Error budget is the allowed failure: 1 - SLO. Interviewers ask: 'How do you define an SLO?' or 'What happens when error budget is exhausted?' Explain that error budget is a tool to balance reliability and feature velocity. When budget is exhausted, development slows down to focus on reliability. Use monitoring tools like Prometheus to track SLIs. Example: latency SLI = p99 latency. In an interview, discuss trade-offs: tighter SLO means less room for error but higher reliability.

# Example: Calculate error budget
slo = 0.999  # 99.9%
error_budget = 1 - slo  # 0.001 (0.1%)
# For a 30-day window:
total_seconds = 30 * 24 * 3600
allowed_failure_seconds = total_seconds * error_budget
print(f"Allowed downtime: {allowed_failure_seconds} seconds ({allowed_failure_seconds/60:.1f} minutes)")

7. Incident Management and Postmortems

SREs handle incidents. Interviewers ask: 'Describe your incident response process.' or 'What is a blameless postmortem?' Explain the stages: detection, response, mitigation, resolution, follow-up. Use tools like PagerDuty for alerting. A blameless postmortem focuses on system failures, not people. Example: 'The deployment caused a config error; we added automated validation.' In an interview, emphasize communication: status updates, escalation paths. Also discuss runbooks: documented procedures for common incidents. Use incident.io or similar.

# Example: Quick incident checklist
# 1. Acknowledge alert
# 2. Assess severity
# 3. Communicate status
# 4. Mitigate (rollback, scale up, etc.)
# 5. Root cause analysis
# 6. Postmortem

echo "Incident response steps:"
echo "1. Acknowledge"
echo "2. Assess"
echo "3. Communicate"
echo "4. Mitigate"
echo "5. RCA"
echo "6. Postmortem"

8. Monitoring and Alerting

Monitoring is key. Interviewers ask: 'What metrics do you monitor?' or 'How do you set up alerts?' Use Prometheus for metrics, Grafana for dashboards, and Alertmanager for alerts. Understand the four golden signals: latency, traffic, errors, saturation. Example: alert on p99 latency > 500ms for 5 minutes. In an interview, discuss alert fatigue: too many alerts cause desensitization. Use proper thresholds and aggregation. Also discuss logging: ELK stack (Elasticsearch, Logstash, Kibana) or Loki. Structured logging helps in debugging.

groups:
  - name: example
    rules:
      - alert: HighLatency
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m])) > 0.5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "High latency detected"