Distributed Locking: The Production Guide to Avoiding Split-Brain and Data Corruption

Distributed locking is one of those things that sounds simple until it breaks your production database at 3 AM. The textbook says 'use a lock' — but the real world says 'your lock just failed and now you have duplicate payments.' This article is the no-bullshit guide to distributed locking: what works, what doesn't, and how to debug when it all goes wrong.

The core problem is mutual exclusion across machines. Without it, two services can simultaneously process the same order, decrement the same inventory twice, or overwrite each other's data. You need a lock that all nodes respect — and that's harder than it sounds because networks are unreliable, clocks drift, and processes crash.

By the end of this, you'll be able to choose the right locking strategy for your system, implement it without the classic mistakes, and diagnose failures when locks misbehave. You'll also know when not to use distributed locking at all — because sometimes the simplest solution is no lock.

Why Distributed Locking Is Hard: The Fallacies of Distributed Computing

Before we talk about solutions, let's talk about why this is a hard problem. The network is not reliable — packets drop, latency spikes, and partitions happen. Clocks are not synchronized — NTP can drift, and even with PTP, you get skew. Processes can pause for garbage collection or get preempted by the OS. These three facts make distributed locking fundamentally different from single-process locking.

Without distributed locking, two nodes can simultaneously modify the same resource. Classic example: an inventory service that decrements stock on order placement. Two orders come in at the same time, both read stock=1, both write stock=0 — you just oversold. The fix is a distributed lock that serializes access to the inventory row.

But here's the kicker: even with a lock, you can still get corruption if the lock expires while the holder is still working. This is the split-brain problem. The holder thinks it has the lock, but another node acquires it and starts modifying the same resource. Now you have two writers. This is why lock fencing (a monotonically increasing token) is critical — it lets the resource reject stale writes.

// io.thecodeforge — System Design tutorial

// Pseudocode for inventory decrement with distributed lock
function decrementStock(orderId, quantity) {
  // Acquire lock on the inventory item
  lock = redisLock.acquire("inventory:item:123", ttl=5000) // 5 second TTL
  if (!lock) {
    throw new Exception("Could not acquire lock, try again")
  }
  try {
    // Read current stock
    stock = db.query("SELECT stock FROM inventory WHERE id=123")
    if (stock < quantity) {
      throw new Exception("Insufficient stock")
    }
    // Update stock
    db.execute("UPDATE inventory SET stock = stock - ? WHERE id=123", quantity)
  } finally {
    // Always release the lock
    redisLock.release(lock)
  }
}

Redis-Based Locks: The Good, the Bad, and the Split-Brain

Redis is the most popular choice for distributed locking because it's fast and simple. The basic pattern: SET key value NX PX 5000 — atomically set the key if it doesn't exist with a 5-second TTL. To release, delete the key only if the value matches your lock token (to avoid deleting someone else's lock).

But Redis has a fundamental problem: it's not strongly consistent. In a Redis cluster, if the master goes down after acknowledging the write but before replicating, the lock is lost. The new master doesn't have the lock, so another client can acquire it. This is why Redis Labs proposed Redlock — a consensus-based algorithm that acquires the lock from a majority of Redis nodes.

Redlock is controversial. Martin Kleppmann famously argued it's unsafe because of clock drift and GC pauses. In practice, it works well if you have tight clock sync (NTP) and short TTLs. But if you need absolute correctness, use ZooKeeper or etcd. For most systems, Redis locks are good enough — just be aware of the edge cases.

# io.thecodeforge — System Design tutorial

import redis
import uuid
import time

class RedisLock:
    def __init__(self, client, key, ttl=5000):
        self.client = client
        self.key = key
        self.ttl = ttl
        self.token = str(uuid.uuid4())  # Unique per lock attempt

    def acquire(self):
        # SET NX PX — atomic: set if not exists, with TTL
        result = self.client.set(self.key, self.token, nx=True, px=self.ttl)
        return result is True

    def release(self):
        # Lua script to atomically delete only if token matches
        lua = """
        if redis.call("get", KEYS[1]) == ARGV[1] then
            return redis.call("del", KEYS[1])
        else
            return 0
        end
        """
        self.client.eval(lua, 1, self.key, self.token)

# Usage
client = redis.Redis(host='localhost', port=6379)
lock = RedisLock(client, "lock:order:123", ttl=5000)
if lock.acquire():
    try:
        # Critical section
        print("Lock acquired, processing order 123")
    finally:
        lock.release()
else:
    print("Failed to acquire lock")

ZooKeeper and etcd: Strong Consistency at a Cost

When correctness matters more than latency, use ZooKeeper or etcd. Both use consensus algorithms (Zab and Raft respectively) to provide linearizable writes. The lock pattern: create an ephemeral sequential node. The client with the smallest sequence number holds the lock. When the client disconnects (or crashes), the ephemeral node is automatically deleted — no TTL needed.

This solves the lock expiration problem because the lock lives as long as the session. But it introduces new problems: session timeouts. If the client's session expires due to a network blip, the lock is released even though the client is still working. This can cause split-brain again. The solution is to use a fencing token: the lock service gives you a monotonically increasing token that you pass to the resource. The resource rejects any write with a stale token.

etcd has a built-in locking package (concurrency/stm) that handles this. ZooKeeper requires more manual work. The trade-off: ZooKeeper/etcd are slower than Redis (10-100ms vs 1ms) but provide stronger guarantees. Use them for critical resources like leader election or financial transactions.

// io.thecodeforge — System Design tutorial

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	clientv3 "go.etcd.io/etcd/client/v3"
	"go.etcd.io/etcd/client/v3/concurrency"
)

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"localhost:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}
	defer cli.Close()

	// Create a session with a 10-second TTL
	session, err := concurrency.NewSession(cli, concurrency.WithTTL(10))
	if err != nil {
		log.Fatal(err)
	}
	defer session.Close()

	// Create a mutex for the resource
	mutex := concurrency.NewMutex(session, "/my-lock")

	// Acquire lock (blocks until acquired)
	if err := mutex.Lock(context.TODO()); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Lock acquired")

	// Critical section
	time.Sleep(2 * time.Second)

	// Release lock
	if err := mutex.Unlock(context.TODO()); err != nil {
		log.Fatal(err)
	}
	fmt.Println("Lock released")
}

Database-Based Locks: The Old Reliable

If you're already using a relational database, you can use row-level locks with SELECT FOR UPDATE. This is the simplest distributed lock — no extra infrastructure. The lock is held for the duration of the transaction. When the transaction commits or rolls back, the lock is released.

But there's a catch: database locks are coarse and can become a bottleneck. If you lock a row for too long, other transactions queue up. Also, if the application crashes mid-transaction, the lock is held until the database detects the dead connection (which can take minutes). This is why you should keep transactions short and use a timeout.

Another pattern: use a database table as a lock registry with a unique constraint on the lock name. INSERT a row with the lock name and a token. If the INSERT succeeds, you have the lock. DELETE to release. This is simple but doesn't handle crashes — you need a background job to clean up stale locks.

-- io.thecodeforge — System Design tutorial

-- Create lock table
CREATE TABLE distributed_locks (
    lock_name VARCHAR(255) PRIMARY KEY,
    token VARCHAR(255) NOT NULL,
    acquired_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    ttl_seconds INT NOT NULL
);

-- Acquire lock: INSERT if not exists, with a TTL check
-- Use ON DUPLICATE KEY UPDATE to extend if same token
INSERT INTO distributed_locks (lock_name, token, ttl_seconds)
VALUES ('order:123', 'token-abc', 30)
ON DUPLICATE KEY UPDATE
    token = IF(acquired_at < NOW() - INTERVAL ttl_seconds SECOND, VALUES(token), token),
    acquired_at = IF(ROW_COUNT() = 0, NOW(), acquired_at);

-- Check if we acquired the lock
SELECT ROW_COUNT() > 0 AS acquired;

-- Release lock
DELETE FROM distributed_locks WHERE lock_name = 'order:123' AND token = 'token-abc';

When Not to Use Distributed Locking

Distributed locking is a hammer, but not every problem is a nail. Sometimes you can avoid locks entirely by using idempotent operations or optimistic concurrency. For example, instead of locking an inventory row, use an atomic decrement: UPDATE inventory SET stock = stock - 1 WHERE stock > 0. If the update affects zero rows, you know stock was insufficient. No lock needed.

Another alternative: use a message queue with exactly-once semantics. Process orders sequentially from a single partition. This avoids locks but introduces ordering constraints. Or use a database transaction with SERIALIZABLE isolation — but that kills performance.

The rule of thumb: if you can design your system to be conflict-free (e.g., using event sourcing or CRDTs), do that instead. Distributed locking should be your last resort, not your first instinct. It adds latency, complexity, and failure modes.

-- io.thecodeforge — System Design tutorial

-- Atomic decrement with check — no lock needed
UPDATE inventory
SET stock = stock - 1
WHERE id = 123 AND stock > 0;

-- Check if the update succeeded
SELECT ROW_COUNT() > 0 AS decremented;