Cache Stampede Prevention — Mutex vs Background Refresh

Every millisecond matters at scale. When Netflix serves 200 million subscribers or Twitter surfaces a trending tweet, a database query that takes 50ms repeated ten thousand times per second will buckle your infrastructure and light your AWS bill on fire. Caching is the single highest-leverage tool in a backend engineer's kit, yet most engineers treat it as an afterthought — a Redis call dropped in after the database is already struggling. The engineers who design caches thoughtfully are the ones who build systems that survive virality.

The core problem caching solves is the impedance mismatch between how fast your application needs data and how fast your storage layer can produce it. Disk-based databases are optimised for durability and complex queries, not for microsecond reads of the same user profile record fifty thousand times per minute. A well-placed cache absorbs that repeated read pressure, serves data from memory at nanosecond speed, and lets your database do what it's actually good at — handling writes and complex aggregations.

By the end of this article you'll be able to walk into a system design interview and explain cache placement strategies, eviction policies and their trade-offs, cache invalidation approaches and their consistency guarantees, how distributed caches like Redis Cluster work internally, and the production failure modes (stampedes, poisoned caches, thundering herds) that separate senior engineers from mid-levels. You'll have working Java code you can reason about, and you'll know exactly what the interviewer is fishing for when they ask 'how would you cache this?'

What a Caching System Interview Actually Tests

A caching system interview evaluates your ability to design a layer that stores frequently accessed data in fast memory (e.g., Redis, Memcached) to reduce latency and load on a primary datastore. The core mechanic is a trade-off: you accept eventual consistency for O(1) reads instead of O(n) database queries. The interviewer wants to see you reason about hit ratios, TTLs, eviction policies (LRU, LFU), and failure modes like thundering herds or cache stampedes.

In practice, you must decide between write-through, write-around, and write-back strategies. Write-through ensures consistency but adds write latency; write-back improves write throughput at the risk of data loss. Key properties that matter are cache invalidation granularity (do you evict a single key or a whole partition?) and the staleness window — how long can a stale value be served before the next refresh? A common benchmark: a 95% hit ratio reduces read latency from 50ms to under 5ms, but a stampede can spike database connections to 10x normal.

Use caching when reads dominate writes and the data changes infrequently — think user sessions, product catalogs, or configuration. It's critical in systems where a single database query costs 100ms+ and the cache can serve in under 1ms. Avoid caching for rapidly mutating data (e.g., real-time stock ticks) unless you accept bounded staleness. In production, a poorly designed cache can cause cascading failures: a single key expiry triggers thousands of concurrent recomputations, overwhelming the database.

Anatomy of a Distributed Cache: More Than Just a Key-Value Store

In an interview, you're not just 'using Redis'; you're designing a high-availability, low-latency data layer. A production-grade caching system consists of three pillars: Storage (usually in-memory Hash Maps or B-Trees), an Eviction Policy to handle capacity limits, and a Consistency Strategy to ensure the cache doesn't serve stale data while the database has moved on.

When we talk about distributed caching, we introduce a fourth pillar: Partitioning (Sharding). Since a single node can't hold all the data or handle all the traffic, we use Consistent Hashing to distribute keys across multiple nodes. This minimizes data movement when a node is added or removed — a critical detail that distinguishes a Senior candidate from a Junior one.

package io.thecodeforge.cache;

import java.util.LinkedHashMap;
import java.util.Map;

/**
 * A production-grade LRU (Least Recently Used) Cache implementation.
 * Using LinkedHashMap to maintain access order for O(1) eviction.
 */
public class SimpleLruCache<K, V> extends LinkedHashMap<K, V> {
    private final int capacity;

    public SimpleLruCache(int capacity) {
        // true for access-order, false for insertion-order
        super(capacity, 0.75f, true);
        this.capacity = capacity;
    }

    @Override
    protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
        // Evict the least recently accessed entry when capacity is exceeded
        return size() > capacity;
    }

    public static void main(String[] args) {
        SimpleLruCache<String, String> cache = new SimpleLruCache<>(3);
        cache.put("user:1", "Alice");
        cache.put("user:2", "Bob");
        cache.put("user:3", "Charlie");
        
        cache.get("user:1"); // user:1 becomes the most recently used
        cache.put("user:4", "David"); // Capacity exceeded, user:2 is evicted

        System.out.println("Cache keys after eviction: " + cache.keySet());
    }
}

Write Strategies: Balancing Speed and Safety

How you update the cache determines your system's consistency.

1. Write-Through: Data is written to the cache and the database simultaneously. High consistency, but adds latency to writes.
2. Write-Around: Data is written only to the database. The cache is only updated on a 'miss'. This prevents the cache from being flooded with data that is rarely read.
3. Write-Back (Write-Behind): Data is written to the cache first, and the database is updated asynchronously. This provides the highest performance but risks data loss if the cache node crashes before the DB is updated.

version: '3.8'
services:
  redis-cache:
    image: redis:7.2-alphine
    container_name: thecodeforge-redis
    ports:
      - "6379:6379"
    command: ["redis-server", "--maxmemory", "512mb", "--maxmemory-policy", "allkeys-lru"]
    networks:
      - forge-network

networks:
  forge-network:
    driver: bridge

Cache Invalidation: The Hardest Problem in Computer Science

There are only two hard things in computer science: cache invalidation and naming things. Invalidation ensures that stale data is removed or updated when the underlying source changes. Common approaches:

1. TTL-based: Set an expiry time. Simple but can serve stale data within the TTL window.
2. Event-based invalidation: Publish a cache invalidation event when the DB is updated (e.g., Redis Pub/Sub, Kafka). Near real-time consistency.
3. Write-through (already covered): Synchronously update cache on write — strong consistency but higher write latency.
4. Read-repair: In distributed caches, if a stale value is read, the reader triggers an update. Used in Dynamo-style systems.

In practice, most systems use a combination: TTL as a safety net, event-based for critical data.

Distributed Cache Topologies: Replication vs Sharding

When a single node can't handle traffic or memory, you need multiple nodes. Two primary patterns:

Replication: Data is copied to multiple nodes (e.g., Redis Sentinel). High read throughput, but write amplification. Writes go to primary, then async to replicas — eventual consistency.

Sharding: Data is partitioned across nodes (e.g., Redis Cluster, Memcached with client-side hashing). Each node owns a subset of keys. Requires consistent hashing to minimize rehashing.

Trade-off: Replication is simpler but wastes memory (each node stores all data). Sharding is memory-efficient but adds complexity for cross-node operations (e.g., multi-key commands fail if keys are on different nodes).

In interviews, start with replication for read-heavy workloads, then scale to sharding when memory of a single node becomes the bottleneck.

Cache Failure Modes: What Breaks Your Cache in Production

Three classic failure modes every senior engineer must explain:

Cache Stampede (Thundering Herd): A hot key expires, thousands of requests try to repopulate it simultaneously, overloading the database. Fix: Use a mutex/lease to allow only one thread to rebuild; others wait. Or use probabilistic early expiration (like Twitter's 'cache roulette').

Cache Penetration: Requests for non-existent keys (e.g., user IDs that don't exist) bypass cache and hit the DB every time. If these are common (e.g., DDOS attacks), the DB gets hammered. Fix: Cache 'null' results with a short TTL (e.g., 1 minute) or use a Bloom filter upfront.

Cache Avalanche: A large number of keys expire at the same time (e.g., all users cached at midnight with same TTL). DB gets a sudden load spike. Fix: Add random jitter to TTLs so they expire at different times. Or use a secondary local cache to absorb the initial misses.

Cache Eviction: Why Your "Hot" Data Stays Cold

You've got a cache. It's full. Something has to go. The eviction policy you choose decides which data survives and which gets nuked. This isn't academic — it's the difference between a snappy API and a thundering herd of cache misses that crush your database.

LRU (Least Recently Used) is the default for most production caches. It kicks out the item nobody touched longest. Works great for workloads with temporal locality — think news feeds or session data. But watch out: one-off batch jobs scanning your cache can poison LRU by touching old entries, making them look "recent" while your real hot data gets evicted.

LFU (Least Frequently Used) tracks access frequency instead of recency. Better for content that's popular consistently but accessed infrequently — like a config file read once per hour. The downside? LFU burns memory tracking counters for every key. Implement it with a min-heap or approximate counts (count-min sketch) to keep overhead sane.

Don't overthink this. Start with LRU. Set a TTL. Monitor your miss ratio. If you see hot keys cycling out, switch to LFU or add a small second-level cache. Your cache isn't smart — you have to feed it the right strategy.

// io.thecodeforge
import java.util.*;

public class LRUCache<K, V> {
    private final int capacity;
    private final LinkedHashMap<K, V> map;

    public LRUCache(int capacity) {
        this.capacity = capacity;
        // accessOrder=true: rearranges on get()
        this.map = new LinkedHashMap<K, V>(capacity, 0.75f, true) {
            @Override
            protected boolean removeEldestEntry(Map.Entry<K, V> eldest) {
                return size() > LRUCache.this.capacity;
            }
        };
    }

    public V get(K key) {
        return map.getOrDefault(key, null);
    }

    public void put(K key, V value) {
        map.put(key, value);
    }
}

Types of Cache: Where You Park Your Hot Data Matters

Every cache lives somewhere. Pick the wrong spot and you're adding latency instead of removing it. Here's the real breakdown.

Application Server Cache lives in-process — right inside your Java heap or Python dict. Fastest possible read (microseconds) because there's no network hop. Problem? You scale horizontally and now each node has its own cache. A request hits node A, caches a user profile. Same user hits node B — cache miss. You burn database queries warming up every new server.

Distributed Cache like Redis or Memcached sits outside your app servers. All nodes share one logical cache. Slower reads (milliseconds) but consistent hit rates. The trade-off: network calls add latency, and your cache becomes a single point of failure unless you shard or replicate. Every production system I've inherited had a Redis cluster that went down at 3 AM — plan for it.

CDN / Edge Cache is for static or semi-static content — images, CSS, API responses with long TTLs. Your users are global but your database is in us-east-1. Edge nodes cache at 200+ locations worldwide. User in Tokyo gets the response from Tokyo edge node — no cross-Pacific round trip. Use this for anything that doesn't change every second.

Global Cache is a hybrid — one logical cache but physically distributed. Usually a write-through to a central node with read replicas close to users. Complex to set up, but necessary for systems like Google or Netflix where consistency and speed both matter.

Rule of thumb: start with application cache for local hot data. Add distributed cache when you need consistency across servers. Add CDN when your users are global and your data is mostly read.

// io.thecodeforge
import redis.clients.jedis.JedisPool;
import redis.clients.jedis.JedisPoolConfig;
import com.github.benmanes.caffeine.cache.Caffeine;

public class CacheTopologyConfig {
    // L1: application cache (Caffeine) — microsecond reads
    public com.github.benmanes.caffeine.cache.Cache<String, String> buildL1() {
        return Caffeine.newBuilder()
                .maximumSize(10_000)
                .expireAfterWrite(5, java.util.concurrent.TimeUnit.MINUTES)
                .build();
    }

    // L2: distributed cache (Redis) — consistent across nodes
    public JedisPool buildL2() {
        JedisPoolConfig poolConfig = new JedisPoolConfig();
        poolConfig.setMaxTotal(50);
        return new JedisPool(poolConfig, "redis-cluster.prod.internal", 6379);
    }
}

Distributed Caching: Redis Cluster vs Memcached vs Hazelcast

Choosing the right distributed cache is critical for performance and scalability. Redis Cluster, Memcached, and Hazelcast are three popular options, each with distinct trade-offs.

Redis Cluster is an in-memory data structure store supporting advanced data types (strings, hashes, lists, sets, sorted sets, bitmaps, HyperLogLog, streams). It offers built-in replication, automatic sharding via hash slots, and high availability with Redis Sentinel. Redis Cluster is ideal for use cases requiring complex data operations, pub/sub messaging, or persistence. However, it has a higher memory footprint and operational complexity compared to Memcached.

Memcached is a simple, high-performance distributed memory object caching system. It supports only string values and has a minimal feature set (no persistence, no replication, no clustering). Memcached excels at caching simple key-value pairs with extremely low latency and high throughput. It uses consistent hashing for distribution, making it easy to scale horizontally. However, it lacks advanced data structures and durability, making it suitable only for transient caching.

Hazelcast is an in-memory data grid (IMDG) that provides distributed data structures (maps, queues, topics) with strong consistency and high availability. It supports partitioning, replication, and near-cache for low-latency access. Hazelcast integrates tightly with Java applications and offers features like distributed computing (executor service), entry processors, and event listeners. It is ideal for applications requiring a distributed data store with compute capabilities. However, it has a steeper learning curve and is primarily Java-centric.

Example scenario: For a high-traffic e-commerce site, Redis Cluster can cache product details with complex queries, Memcached can cache session data, and Hazelcast can manage distributed counters for inventory.

Decision factors: Consider data complexity (Redis for complex, Memcached for simple), persistence needs (Redis/Hazelcast), latency requirements (Memcached lowest), and ecosystem fit (Hazelcast for Java).

# Example: Connecting to each cache
import redis
import pymemcache
import hazelcast

# Redis Cluster
rc = redis.RedisCluster(host='localhost', port=7000)
rc.set('key', 'value')
print(rc.get('key'))

# Memcached
mc = pymemcache.Client(('localhost', 11211))
mc.set('key', 'value')
print(mc.get('key'))

# Hazelcast
client = hazelcast.HazelcastClient()
map = client.get_map('my-distributed-map').blocking()
map.put('key', 'value')
print(map.get('key'))

Cache Eviction Policies: LFU, LRU, FIFO, ARC Comparison

Cache eviction policies determine which entries to remove when the cache reaches capacity. The choice impacts hit rate and performance.

LRU (Least Recently Used) evicts the entry that has not been accessed for the longest time. It assumes that recently accessed data is likely to be accessed again. LRU is simple to implement and works well for workloads with temporal locality. However, it can be vulnerable to scan attacks where a one-time scan evicts hot data.

LFU (Least Frequently Used) evicts the entry with the lowest access frequency. It retains frequently accessed data, making it suitable for workloads with skewed access patterns. However, LFU can suffer from cache pollution by old data that was once popular but is no longer relevant. It also has higher overhead due to maintaining frequency counters.

FIFO (First In, First Out) evicts the oldest entry regardless of access patterns. It is simple and fair but performs poorly for workloads with temporal locality, as it may evict frequently accessed data that has been in the cache for a long time.

ARC (Adaptive Replacement Cache) dynamically balances between LRU and LFU by maintaining two lists: recent and frequent. It adapts to changing workloads, providing better hit rates than LRU or LFU alone. ARC is more complex but offers robust performance across diverse access patterns.

Example: For a social media feed, LRU works well for recent posts, LFU for viral content, FIFO for session data, and ARC for mixed workloads.

Implementation considerations: LRU can be implemented with a doubly linked list and hash map (O(1) operations). LFU requires a min-heap or frequency list. ARC uses two LRU lists and a ghost list for history.

Trade-offs: LRU is simple but scan-resistant; LFU retains hot data but can be stale; FIFO is fair but inefficient; ARC adapts but is complex.

from collections import OrderedDict

class LRUCache:
    def __init__(self, capacity: int):
        self.cache = OrderedDict()
        self.capacity = capacity

    def get(self, key: int) -> int:
        if key not in self.cache:
            return -1
        self.cache.move_to_end(key)
        return self.cache[key]

    def put(self, key: int, value: int) -> None:
        if key in self.cache:
            self.cache.move_to_end(key)
        self.cache[key] = value
        if len(self.cache) > self.capacity:
            self.cache.popitem(last=False)

Cache-aside vs Read-through vs Write-through: Decision Guide

Caching strategies define how data flows between the application, cache, and database. The three main patterns are cache-aside, read-through, and write-through.

Cache-aside (Lazy Loading): The application is responsible for loading data into the cache. On a cache miss, the application fetches data from the database, stores it in the cache, and returns it. On updates, the application invalidates the cache entry (or updates it). This pattern gives the application full control over caching logic. It is simple and works well for read-heavy workloads with infrequent updates. However, it can lead to stale data if invalidation is missed, and it adds latency on cache misses.

Read-through: The cache itself loads data from the database on a miss. The application only interacts with the cache. This simplifies application code and ensures consistency between cache and database (if the cache handles invalidation). However, it requires the cache to support read-through (e.g., Redis with RedisJSON, or custom implementation). It can also cause cache stampedes if many requests miss simultaneously.

Write-through: The application writes data to both cache and database synchronously. This ensures strong consistency between cache and database. However, it adds write latency and can reduce throughput. It is suitable for write-heavy workloads where data must be immediately consistent.

Example: For a product catalog with frequent reads and occasional updates, cache-aside with lazy loading is common. For a session store where consistency is less critical, read-through may suffice. For a financial system, write-through ensures no stale data.

Combined patterns: You can mix patterns, e.g., read-through for reads and write-behind for writes (asynchronous updates).

import redis
import psycopg2

cache = redis.Redis()
db = psycopg2.connect("dbname=test")

def get_user(user_id):
    # Cache-aside
    user = cache.get(f"user:{user_id}")
    if user is None:
        cursor = db.cursor()
        cursor.execute("SELECT * FROM users WHERE id = %s", (user_id,))
        user = cursor.fetchone()
        cache.setex(f"user:{user_id}", 3600, user)
    return user

def update_user(user_id, data):
    # Update database
    cursor = db.cursor()
    cursor.execute("UPDATE users SET ... WHERE id = %s", (user_id,))
    db.commit()
    # Invalidate cache
    cache.delete(f"user:{user_id}")