System Design Interview - Cache Stampede in Production

Every senior engineering role at a top tech company has one brutal filter: the system design round. It's the interview that makes experienced developers freeze up, not because they lack knowledge, but because the question is deliberately open-ended. 'Design Twitter.' 'Design a URL shortener.' 'Design Netflix.' The candidate who answers these well isn't the one who memorized the most blog posts — it's the one who can think out loud, reason through trade-offs, and communicate at the level of a staff engineer.

The core problem this interview type solves — from the interviewer's perspective — is figuring out how you'll behave when given an ambiguous, high-stakes technical problem with no single right answer. At scale, every architectural decision has cascading consequences. Choosing the wrong database engine, ignoring read/write ratios, or failing to think about failure modes can mean millions in lost revenue or a 3am outage. The design interview is a compressed simulation of exactly that situation.

By the end of this guide, you'll have a repeatable framework you can apply to any system design prompt, understand the specific trade-offs interviewers are listening for (and the buzzwords that actually hurt you), know how to handle the moments where you genuinely don't know the answer, and walk away with a mental model that works in real production systems — not just whiteboards.

The 4-Step Framework for Architectural Clarity

A System Design Interview isn't a coding test; it's a conversation about trade-offs. To avoid the 'blank whiteboard' syndrome, you need a reliable framework. We recommend the following: 1. Understand Requirements (Functional & Non-Functional), 2. High-Level Design (The 'Boxes and Arrows'), 3. Deep Dive into Bottlenecks (Database Sharding, Caching), and 4. Wrap-up (Identifying SPOFs and Scaling).

Rather than starting with a dry definition, let's see a practical example of how you might handle a request-response flow in a distributed environment.

Here's the thing most candidates miss: the framework is just a container. What matters is how you move through it. Don't treat it as a rigid checklist — treat it as a conversation. If the interviewer asks a deep question about caching mid-way through step 2, follow that thread. The framework keeps you from getting lost, not from getting interesting.

package io.thecodeforge.design;

import java.util.concurrent.atomic.AtomicLong;

/**
 * A production-grade concept for Unique ID Generation in a distributed system.
 * In a real interview, you'd discuss Snowflake IDs or UUIDs to avoid DB bottlenecks.
 */
public class DistributedIdGenerator {
    private final long datacenterId;
    private final long workerId;
    private final AtomicLong sequence = new AtomicLong(0L);

    public DistributedIdGenerator(long datacenterId, long workerId) {
        this.datacenterId = datacenterId;
        this.workerId = workerId;
    }

    public synchronized String generateId() {
        long timestamp = System.currentTimeMillis();
        // In a real interview, explain how bit-shifting ensures sortability and uniqueness
        return String.format("%d-%d-%d-%d", timestamp, datacenterId, workerId, sequence.getAndIncrement());
    }

    public static void main(String[] args) {
        DistributedIdGenerator generator = new DistributedIdGenerator(1, 42);
        System.out.println("Generated Unique ID: " + generator.generateId());
    }
}

Infrastructure as Code: Deploying the Architecture

Interviewers love when you can bridge the gap between a whiteboard drawing and actual deployment. Understanding how to containerize and scale your components is key to demonstrating seniority.

But don't just name tools. Explain why you'd choose Docker over a VM, or Kubernetes over a single server. The 'how' is easy — the 'why' is what separates staff engineers.

The dirty secret: most senior engineers have been burned by overcomplicated infrastructure. If you're interviewing, showing you understand the pain of maintaining a 15-microservice nightclub is more impressive than listing every AWS service. Say 'I'd start with a monolith until I have a concrete bottleneck.' That's the answer that gets 'hire.'

# TheCodeForge - Scaling the API Layer
FROM eclipse-temurin:17-jdk-alpine

# Best practice: Don't run as root in production interview scenarios
RUN addgroup -S forgegroup && adduser -S forgeuser -G forgegroup
USER forgeuser

WORKDIR /app
COPY target/system-design-app.jar app.jar

# Expose the service port
EXPOSE 8080

# Standard entrypoint for Spring Boot apps
ENTRYPOINT ["java", "-Xmx2g", "-jar", "app.jar"]

Back-of-Envelope Estimations: The Numbers That Validate Your Design

In a system design interview, you can talk theoretical all day, but the moment you put numbers on the whiteboard, you demonstrate real-world experience. Interviewers want to see you can estimate QPS, storage, bandwidth, and cache size with reasonable accuracy.

Don't aim for precision — aim for orders of magnitude. A factor of 10 off is acceptable if you catch it and adjust. Here's the cheat sheet: 1 million requests/second = 1,000,000 QPS. 1 TB = 1000 GB. A single MySQL write can handle ~1k writes/second. A Redis single node can handle ~100k reads/second.

Here's a real trick: always mention 'I'd add 30% headroom for traffic spikes.' It shows you've dealt with production surprises. If you forget to include replication factor (3x) and retention (say 18 months), your storage estimate will be off by a factor of 5 — and your architecture will be wrong.

from typing import Tuple

def estimate_qps(daily_active_users: int, requests_per_user: int, peak_multiplier: float = 2.0) -> Tuple[float, float]:
    """Estimate average and peak QPS for a system."""
    daily_requests = daily_active_users * requests_per_user
    avg_qps = daily_requests / 86400
    peak_qps = avg_qps * peak_multiplier
    return round(avg_qps, 2), round(peak_qps, 2)

# Example for Twitter-like system: 100M DAU, each user fetches timeline 50 times
avg, peak = estimate_qps(100_000_000, 50)
print(f"Average QPS: {avg}, Peak QPS: {peak}")
# Output: Average QPS: 57870.37, Peak QPS: 115740.74

Trade-Off Decision Tree: SQL vs NoSQL, Cache vs DB, Sync vs Async

Every system design interview forces you to make choices. The ability to articulate the trade-offs is what separates a good answer from a great one. Let's create a mental decision tree:

• If your system requires strong consistency (e.g., banking), go SQL with a single writer and read replicas. Accept write latency.
• If you need high availability with eventual consistency for a global feed, choose NoSQL (Cassandra, DynamoDB).
• If you have a read-heavy workload (90% reads), add a cache layer (Redis) and consider CDN for static assets.
• If your workload is write-heavy (50%+ writes), you need a commit log (Kafka) and a write-optimised store (Cassandra).

The decision tree is not about picking the right answer — it's about showing you understand the constraints.

Don't forget to mention the hard part: consistency trade-offs. If you pick eventual consistency, say 'I accept that a user might see stale data for a few seconds — that's fine for this use case.' If you pick strong consistency, say 'I'm trading availability for correctness — I'll need to handle higher write latency and potential downtime during partitions.' That's the level of depth that gets 'strong hire.'

Failure Modes: Designing for When Things Go Wrong

The most overlooked part of system design interviews is discussing failure modes. Too many candidates describe a perfect system where everything works. The real world has partial failures — a network partition, a node crash, a buggy deployment. A senior engineer designs for these.

Don't just list them — explain the specific scenario. Say 'If the cache cluster goes down, I'd have a circuit breaker that falls back to the database with a pool limiter to prevent overload. The users would see slightly slower responses, but no outage.' That's a senior answer.

import time
from typing import Callable, Any

class CircuitBreaker:
    def __init__(self, max_failures: int = 5, reset_timeout: float = 60.0):
        self.max_failures = max_failures
        self.reset_timeout = reset_timeout
        self.failure_count = 0
        self.last_failure_time = 0.0
        self.state = "CLOSED"  # CLOSED, OPEN, HALF_OPEN

    def call(self, func: Callable, *args, **kwargs) -> Any:
        if self.state == "OPEN":
            if time.time() - self.last_failure_time > self.reset_timeout:
                self.state = "HALF_OPEN"
            else:
                raise Exception("Circuit breaker is OPEN. Request blocked.")
        try:
            result = func(*args, **kwargs)
            if self.state == "HALF_OPEN":
                self.state = "CLOSED"
                self.failure_count = 0
            return result
        except Exception:
            self.failure_count += 1
            self.last_failure_time = time.time()
            if self.failure_count >= self.max_failures:
                self.state = "OPEN"
            raise

# Usage
cb = CircuitBreaker(max_failures=3, reset_timeout=30)
cb.call(self.api.get_timeline, "user123")

The Wrap-Up: Single Points of Failure and Scaling Plans

The final step of your interview answer should be a structured wrap-up. Summarise what you've designed, then explicitly call out: 1. Single points of failure: 'Our load balancer is a SPOF. I'd make it redundant with active-passive and a floating IP.' 2. Scaling plan: 'Currently this handles 1M DAU. To reach 100M DAU, I'd shard the database by user ID, add a Redis cache for timelines, and introduce Kafka for async processing of tweets.' 3. Future improvements: things you'd add if time allowed (like telemetry, rate limiting, etc.)

This structured wrap-up leaves a strong final impression — it tells the interviewer you think holistically.

Pro tip: end with an open-ended question. 'Is there any requirement I missed that would change this design?' That shows you're not attached to your answer — you're collaborating.

Data Partitioning and Sharding Strategies: Consistent Hashing, Range Partitioning, and Rebalancing

When you outgrow a single database instance, partitioning becomes inevitable. The two most common strategies are range partitioning and consistent hashing. Range partitioning splits data by key ranges (e.g., user ID 1-1000 on shard A, 1001-2000 on shard B). It's simple and supports range queries, but hot keys can overload a single shard. Consistent hashing distributes data across a ring using a hash function. It minimizes data movement when nodes join or leave, but range queries become expensive because data is scattered.

In interviews, you need to choose based on query patterns. If you frequently query by user ID range, range partitioning is natural. If you need uniform load and dynamic scaling, consistent hashing wins. Many production systems use a hybrid: consistent hashing with virtual nodes (replicas) to spread load evenly, and secondary indexes for range queries.

Always discuss rebalancing: adding new nodes in a range-partitioned system requires splitting ranges and migrating data. Consistent hashing only moves keys within the affected segment. Tools like Cassandra handle rebalancing automatically using virtual nodes and hinted handoff.

Here's the real-world truth: rebalancing is where most designs break. Say 'I'd use consistent hashing with 150 virtual nodes per physical node to distribute load evenly.' That level of detail signals you've done this before.

package io.thecodeforge.design;

import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.*;

public class ConsistentHashRing<T> {
    private final TreeMap<Long, T> ring = new TreeMap<>();
    private final MessageDigest md;
    private final int replicas;

    public ConsistentHashRing(int replicas) throws NoSuchAlgorithmException {
        this.md = MessageDigest.getInstance("MD5");
        this.replicas = replicas;
    }

    public void addNode(T node) {
        for (int i = 0; i < replicas; i++) {
            long hash = hash(node.toString() + i);
            ring.put(hash, node);
        }
    }

    public void removeNode(T node) {
        for (int i = 0; i < replicas; i++) {
            long hash = hash(node.toString() + i);
            ring.remove(hash);
        }
    }

    public T getNode(String key) {
        if (ring.isEmpty()) return null;
        long hash = hash(key);
        Map.Entry<Long, T> entry = ring.ceilingEntry(hash);
        if (entry == null) entry = ring.firstEntry();
        return entry.getValue();
    }

    private long hash(String key) {
        md.reset();
        md.update(key.getBytes());
        byte[] digest = md.digest();
        return ((long) (digest[3] & 0xFF) << 24) | ((long) (digest[2] & 0xFF) << 16) |
               ((long) (digest[1] & 0xFF) << 8) | (digest[0] & 0xFF);
    }

    public static void main(String[] args) throws Exception {
        ConsistentHashRing<String> ring = new ConsistentHashRing<>(3);
        ring.addNode("shard1");
        ring.addNode("shard2");
        ring.addNode("shard3");
        System.out.println(ring.getNode("user1234"));
    }
}

Observability and Monitoring: Logging, Metrics, and Distributed Tracing

Most system design descriptions skip observability, but in production you can't fix what you can't see. The three pillars are logging, metrics, and distributed tracing. Logs give you per-request detail but are expensive to store long-term. Metrics (latency, error rates, throughput) give you aggregated health. Distributed tracing connects a single request across multiple services.

In an interview, mentioning you'd integrate Prometheus for metrics, structured logging (JSON), and Jaeger for tracing shows operational maturity. Discuss how you'd monitor the key SLIs: latency (P50, P95, P99), error rate, throughput, and saturation (e.g., CPU, memory, connection pool). Define SLOs and set up alerts based on burn-rate budgets.

A common mistake is to design for zero latency but not instrument to measure it. Add a simple tracing middleware from day one. Without tracing, debugging a 500ms latency spike across 10 services becomes a guessing game.

Here's a concrete snippet: 'I'd set up structured logging with correlation IDs, then pipe logs into a centralised system (ELK). For metrics, I'd expose endpoints for Prometheus and create dashboards showing P99 latency, error rate, and throughput. For tracing, I'd use OpenTelemetry with Jaeger.' That's a senior-level answer.

package io.thecodeforge.observability;

import io.opentracing.Span;
import io.opentracing.Tracer;
import io.opentracing.util.GlobalTracer;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import java.io.IOException;

public class TracingMiddleware implements Filter {
    @Override
    public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
            throws IOException, ServletException {
        Tracer tracer = GlobalTracer.get();
        Span span = tracer.buildSpan("api-call")
                .withTag("method", ((HttpServletRequest) request).getMethod())
                .withTag("path", ((HttpServletRequest) request).getRequestURI())
                .start();
        try (Scope ignored = tracer.activateSpan(span)) {
            chain.doFilter(request, response);
        } catch (Exception e) {
            span.setTag("error", true);
            span.log(e.getMessage());
            throw e;
        } finally {
            span.finish();
        }
    }
    // ... other filter methods
}

Idempotency and Retry Strategies: Preventing Double Charges and Data Corruption

In distributed systems, network failures are inevitable. A client sends a request, the server processes it, but the acknowledgment is lost. The client retries — and suddenly you have two orders, two payments, two emails. That's why idempotency is not optional.

Idempotency means performing the same operation multiple times produces the same result. For write operations, use an idempotency key: a unique token generated by the client and sent with the request. The server stores the result keyed by that token; if it sees the same key again, it returns the stored response without executing the operation again.

In interviews, mention idempotency early. Say 'Every write operation will include an idempotency key. The client generates a UUID, sends it with the request. The server deduplicates by that key.' This signals you've built payment systems.

For retries, use exponential backoff with jitter — never retry instantly. A naive retry can bring down a struggling service. Say 'I'd use exponential backoff with a base of 1 second, doubling each time, capped at 30 seconds, plus random jitter (±20%) to spread retries.'

import uuid
from flask import Flask, request, jsonify
from functools import wraps

app = Flask(__name__)
idempotency_store = {}  # In production, use Redis with TTL

def idempotent(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        key = request.headers.get('Idempotency-Key')
        if not key:
            return jsonify({'error': 'Missing Idempotency-Key header'}), 400
        if key in idempotency_store:
            return jsonify(idempotency_store[key]), 200
        response = f(*args, **kwargs)
        idempotency_store[key] = response.get_json()
        return response
    return decorated

@app.route('/order', methods=['POST'])
@idempotent
def create_order():
    # Process order (deduplicated by idempotency key)
    return jsonify({'order_id': str(uuid.uuid4()), 'status': 'created'}), 201