Distributed Caching: How to Stop Your Database From Begging for Mercy

Your database is screaming. It's 2 AM, traffic spiked, and your primary DB is at 100% CPU serving the same 10,000 rows over and over. You've got a cache — a single Redis instance — but it's also melting because every request hits it for the same keys. You need distributed caching. Not because it's trendy, but because your architecture is begging for it.

Distributed caching solves a specific problem: your data access pattern has a hot set that exceeds what one machine can handle. It spreads the load across multiple cache nodes, so no single node becomes the bottleneck. It also gives you fault tolerance — when a node dies, you don't lose the entire cache.

By the end of this, you'll be able to design a distributed cache that survives traffic spikes, node failures, and even your own deployment mistakes. You'll know when to use Redis Cluster vs. Memcached, how consistent hashing actually works, and the exact failure modes that will bite you in production.

Why Your Single Redis Instance Won't Cut It

You've outgrown single-node caching. The problem isn't just capacity — it's throughput. A single Redis instance can handle ~100K ops/sec on a good day. When your traffic demands 500K ops/sec, you need to split the load. But naive sharding (like key % N) is a trap. When you add or remove a node, almost every key remaps to a different server. That means a cache storm — all clients miss simultaneously and hit the database. I've seen this take down a production system at 3 AM. The fix is consistent hashing, which minimizes remapping when the cluster changes.

// io.thecodeforge — System Design tutorial

// Consistent hashing with virtual nodes
// Each physical node gets multiple virtual nodes on the hash ring
// to distribute load evenly.

import java.util.*;

class ConsistentHash<T> {
    private final int numberOfReplicas;
    private final SortedMap<Integer, T> circle = new TreeMap<>();

    public ConsistentHash(int numberOfReplicas, Collection<T> nodes) {
        this.numberOfReplicas = numberOfReplicas;
        for (T node : nodes) {
            add(node);
        }
    }

    public void add(T node) {
        // Add virtual nodes for each physical node
        for (int i = 0; i < numberOfReplicas; i++) {
            circle.put(hash(node.toString() + i), node);
        }
    }

    public void remove(T node) {
        for (int i = 0; i < numberOfReplicas; i++) {
            circle.remove(hash(node.toString() + i));
        }
    }

    public T get(Object key) {
        if (circle.isEmpty()) return null;
        int hash = hash(key);
        // Find the first node clockwise from the key's hash
        if (!circle.containsKey(hash)) {
            SortedMap<Integer, T> tailMap = circle.tailMap(hash);
            hash = tailMap.isEmpty() ? circle.firstKey() : tailMap.firstKey();
        }
        return circle.get(hash);
    }

    private int hash(Object key) {
        // Use a good hash function like Murmur3 or FNV-1a
        return key.hashCode() & 0x7fffffff; // Simple for demo, use better in prod
    }
}

// Usage: ConsistentHash<String> cacheNodes = new ConsistentHash<>(200, nodes);

Cache Topologies: Side Cache vs. Read-Through vs. Write-Through

How your application talks to the cache matters as much as the cache itself. The most common pattern is side cache (aka cache-aside): your app checks cache first, on miss it loads from DB and populates cache. Simple, but you have to handle stale data yourself. Read-through cache (like Redis with a custom module or a proxy) hides the DB call behind the cache — your app just asks the cache, and the cache fetches from DB if missing. Write-through cache updates the cache and DB in the same transaction — ensures consistency but adds latency. Write-behind (write-back) updates cache immediately and DB asynchronously — fast writes but risk of data loss on crash. I've used write-behind for a high-throughput analytics pipeline where losing a few seconds of data was acceptable. For a payments system? Never. Use write-through or side cache with strong consistency checks.

// io.thecodeforge — System Design tutorial

// Cache-aside (side cache) pattern for a checkout service
// Uses Redis for caching user cart data.

import redis.clients.jedis.Jedis;
import com.google.gson.Gson;

public class CartService {
    private final Jedis cache;
    private final Database db;
    private final Gson gson = new Gson();
    private static final String CART_PREFIX = "cart:";
    private static final int TTL_SECONDS = 3600; // 1 hour

    public CartService(Jedis cache, Database db) {
        this.cache = cache;
        this.db = db;
    }

    public Cart getCart(String userId) {
        String key = CART_PREFIX + userId;
        String cached = cache.get(key);
        if (cached != null) {
            return gson.fromJson(cached, Cart.class);
        }
        // Cache miss — load from DB
        Cart cart = db.loadCart(userId);
        if (cart != null) {
            // Populate cache with TTL to avoid stale data
            cache.setex(key, TTL_SECONDS, gson.toJson(cart));
        }
        return cart;
    }

    public void updateCart(String userId, Cart cart) {
        // Write-through: update DB first, then invalidate cache
        db.saveCart(userId, cart);
        cache.del(CART_PREFIX + userId);
        // Next read will miss and fetch fresh data
    }
}

// Output: When getCart is called, if cache miss, DB is queried and cache is populated.
// On update, cache is invalidated to ensure consistency.

Eviction Policies: What Happens When Memory Runs Out

Your cache has finite memory. When it's full, something has to go. The default in Redis is noeviction — it just returns errors on writes. That's a production disaster waiting to happen. You must set maxmemory-policy. The most common is allkeys-lru: evict the least recently used key regardless of TTL. For caches with TTLs, volatile-lru evicts only keys with TTL set. allkeys-lfu (least frequently used) works well for workloads with skewed access patterns. I once saw a team use noeviction on a Redis instance that stored session data. When memory filled up, users couldn't log in. The fix was adding maxmemory 4gb and maxmemory-policy allkeys-lru. Also, monitor eviction rate via INFO command — if it's high, your cache is too small or your TTLs are too long.

# io.thecodeforge — System Design tutorial

# Redis configuration for production cache
# Set in redis.conf or via CONFIG SET

maxmemory 4gb
maxmemory-policy allkeys-lru
# Also consider:
# maxmemory-policy volatile-lru  # only evict keys with TTL
# maxmemory-policy allkeys-lfu   # evict least frequently used

# Monitor evictions:
# redis-cli INFO stats | grep evicted_keys
# If evicted_keys per second > 100, increase maxmemory or reduce TTLs.

Consistency and Stale Data: The CAP Trade-Off

Distributed caches are eventually consistent by nature. When you update data in the DB, the cache might serve stale data for a while. How stale? That depends on your invalidation strategy. The simplest approach is TTL-based expiry: set a reasonable TTL (e.g., 5 minutes) and accept that data can be up to 5 minutes stale. For stricter consistency, use write-through or publish-subscribe invalidation (e.g., Redis Pub/Sub to broadcast cache invalidation events). But even then, there's a window between DB write and cache invalidation where a read could get stale data. In practice, for most systems, a short TTL (seconds to minutes) is good enough. For systems that can't tolerate any staleness (like inventory in an e-commerce checkout), you might skip the cache entirely for that data or use a distributed lock to synchronize reads and writes. I've seen a booking system that cached seat availability with a 30-second TTL — double bookings happened. They switched to a write-through cache with Redis transactions (MULTI/EXEC) to atomically update DB and cache.

// io.thecodeforge — System Design tutorial

// Write-through with Redis transaction for seat booking
// Ensures cache and DB are updated atomically.

import redis.clients.jedis.Jedis;
import redis.clients.jedis.Transaction;

public class BookingService {
    private final Jedis cache;
    private final Database db;

    public boolean bookSeat(String eventId, String seatId, String userId) {
        String cacheKey = "seat:" + eventId + ":" + seatId;
        // Watch the key for optimistic locking
        cache.watch(cacheKey);
        String currentOwner = cache.get(cacheKey);
        if (currentOwner != null) {
            // Seat already booked (in cache)
            return false;
        }
        // Attempt to book in DB
        boolean dbSuccess = db.bookSeat(eventId, seatId, userId);
        if (!dbSuccess) {
            cache.unwatch();
            return false;
        }
        // Update cache atomically
        Transaction t = cache.multi();
        t.set(cacheKey, userId);
        t.expire(cacheKey, 3600); // 1 hour TTL
        List<Object> results = t.exec();
        if (results == null) {
            // Transaction failed (key was modified by another client)
            // Rollback DB? Or handle conflict
            return false;
        }
        return true;
    }
}

// Output: Returns true if booking succeeded. Cache and DB are consistent.

When Not to Use Distributed Caching

Distributed caching isn't free. It adds complexity: network latency (even localhost has overhead), serialization costs, and operational burden (monitoring, backups, cluster management). Don't use it if your entire dataset fits in memory on a single node and your traffic is moderate. A single Redis instance with 32GB RAM can handle most small-to-medium applications. Also, don't cache data that changes every millisecond — the cache invalidation overhead will kill you. For example, real-time stock prices: by the time you cache and serve, the price is stale. Better to use a streaming system. Finally, if your read-to-write ratio is close to 1:1, caching might not help much — you're just adding latency to every write. Measure your actual access patterns before adding a cache layer.