Consistency Models in Distributed Systems: Pick the Right Trade-Off Before Your Data Breaks

// io.thecodeforge — System Design tutorial

// Simulating CAP trade-off in a distributed key-value store
// Scenario: two nodes, network partition between them

// Node A receives a write: key="cart:user1", value="item123"
// Node B is partitioned, cannot receive the write

// If we choose CP (strong consistency):
//   - Node A refuses the write until partition heals
//   - Result: write fails, but data is consistent

// If we choose AP (eventual consistency):
//   - Node A accepts the write locally
//   - When partition heals, nodes reconcile
//   - Result: write succeeds, but Node B may serve stale reads

// In production, you often mix models per operation.
// Example: DynamoDB allows per-request consistency choice.

// Strong read:
// GetItemRequest request = new GetItemRequest()
//     .withTableName("Orders")
//     .withKey(Collections.singletonMap("OrderId", new AttributeValue("123")))
//     .withConsistentRead(true);  // Forces read from leader, higher latency

// Eventual read (default):
// GetItemRequest request = new GetItemRequest()
//     .withTableName("Orders")
//     .withKey(Collections.singletonMap("OrderId", new AttributeValue("123")))
//     .withConsistentRead(false); // May read stale data, lower latency

// io.thecodeforge — System Design tutorial

// Simulating a strongly consistent write in a Raft-based system
// Assume 3 nodes: leader, follower1, follower2

// Client sends write to leader
// Leader appends to its log, sends AppendEntries to followers
// Follower1 acknowledges, follower2 is slow (network latency)
// Leader waits for majority (2/3) acknowledgment
// Once majority acknowledged, leader commits and responds to client

// Pseudocode for leader write path:
// function handleWrite(key, value) {
//     logEntry = {term: currentTerm, index: nextIndex++, key, value}
//     appendToLog(logEntry)
//     sendAppendEntriesToFollowers(logEntry)
//     // Wait for majority acknowledgment with timeout
//     if (majorityAcknowledged(timeout=100ms)) {
//         applyToStateMachine(key, value)
//         return SUCCESS
//     } else {
//         return TIMEOUT  // Client must retry
//     }
// }

// Client read: must contact leader to ensure latest committed value
// function handleRead(key) {
//     if (currentRole != LEADER) {
//         redirectToLeader()
//     }
//     return stateMachine.get(key)  // Already committed
// }

// Trade-off: if follower2 is partitioned, leader cannot commit writes
// System becomes unavailable until partition heals

// io.thecodeforge — System Design tutorial

// Simulating eventual consistency in Cassandra
// Keyspace with replication factor 3, consistency level ONE for writes and reads

// Write: key="balance:user42", value=100
// Coordinator sends write to one replica (consistency ONE)
// Replica A acknowledges, write returns success
// Replicas B and C are not updated yet (asynchronous replication)

// Read: immediate read after write
// Coordinator sends read to one replica (consistency ONE)
// If it hits replica B or C, it returns stale value (e.g., 50)
// Eventually, anti-entropy or read-repair will update B and C

// In production, use QUORUM for critical data:
// Write: consistency QUORUM (2 out of 3 replicas must acknowledge)
// Read: consistency QUORUM (2 out of 3 replicas must respond, compare timestamps)
// This ensures strong consistency at the cost of latency

// Example with DataStax driver:
// Statement write = new SimpleStatement("UPDATE balances SET amount = ? WHERE user_id = ?")
//     .setConsistencyLevel(ConsistencyLevel.QUORUM);
// session.execute(write);

// Statement read = new SimpleStatement("SELECT amount FROM balances WHERE user_id = ?")
//     .setConsistencyLevel(ConsistencyLevel.QUORUM);
// ResultSet rs = session.execute(read);

// io.thecodeforge — System Design tutorial

// Simulating causal consistency with version vectors
// Each operation carries a vector clock: [node1_version, node2_version, ...]

// User A posts comment (operation X) on node1
// Vector clock after X: [1, 0, 0]

// User A likes comment (operation Y) on node2
// Y must happen after X, so Y's vector clock must include X's clock
// Y's vector clock: [1, 1, 0] (node1 version=1, node2 version=1)

// Node3 receives Y before X
// Node3 checks Y's dependencies: requires node1 version >=1
// Node3's node1 version is 0, so it buffers Y
// When X arrives, node3 applies X, then applies Y

// In production, this is complex. Most databases don't implement it fully.
// Instead, use session consistency (read-your-writes) which is simpler.

// Example with Riak (supports causal consistency via vector clocks):
// // Write with vector clock
// StoreValue sv = new StoreValue.Builder("value")
//     .withVectorClock(previousClock)
//     .build();
// client.store(sv).execute();

// // Read returns vector clock for next write
// FetchValue fv = new FetchValue.Builder("key").build();
// FetchValue.Response response = client.fetch(fv).execute();
// VectorClock clock = response.getVectorClock();

// io.thecodeforge — System Design tutorial

// Implementing read-your-writes in a custom distributed store
// Client maintains a write timestamp (lastWriteTimestamp)

// Write:
// function write(key, value) {
//     timestamp = getCurrentTimestamp()
//     // Send write to all replicas with timestamp
//     for (replica in replicas) {
//         replica.write(key, value, timestamp)
//     }
//     client.lastWriteTimestamp = timestamp
// }

// Read:
// function read(key) {
//     // Read from a replica that has seen our write
//     // Option 1: read from all replicas, pick the one with highest timestamp
//     // Option 2: use a session token that identifies a replica
//     replica = getReplicaWithTimestampAtLeast(client.lastWriteTimestamp)
//     return replica.read(key)
// }

// In practice, databases handle this:
// - MongoDB: use "majority" read concern
// - PostgreSQL: use synchronous replication
// - Redis: use WAIT command to ensure replication

// Example with MongoDB driver:
// MongoCollection<Document> collection = database.getCollection("users");
// // Write with write concern majority
// collection.insertOne(document, new InsertOneOptions().writeConcern(WriteConcern.MAJORITY));
// // Read with read concern majority
// FindIterable<Document> docs = collection.find()
//     .readConcern(ReadConcern.MAJORITY);

// io.thecodeforge — System Design tutorial

// E-commerce checkout service with mixed consistency

// 1. Check inventory (strong consistency)
// Must read the latest inventory count to avoid overselling
// function checkInventory(productId, quantity) {
//     // Use strong read (e.g., QUORUM in Cassandra)
//     int available = inventoryDB.read(productId, ConsistencyLevel.QUORUM);
//     return available >= quantity;
// }

// 2. Reserve inventory (strong consistency)
// Must atomically decrement inventory
// function reserveInventory(productId, quantity) {
//     // Use compare-and-set or conditional update
//     boolean success = inventoryDB.update(
//         "UPDATE inventory SET count = count - ? WHERE product_id = ? AND count >= ?",
//         quantity, productId, quantity
//     );
//     return success;
// }

// 3. Add to cart (read-your-writes)
// User must see their own cart updates
// function addToCart(userId, productId) {
//     cartDB.write(userId, productId, ConsistencyLevel.LOCAL_QUORUM);
//     // Read back with same consistency
//     return cartDB.read(userId, ConsistencyLevel.LOCAL_QUORUM);
// }

// 4. Get recommendations (eventual consistency)
// function getRecommendations(userId) {
//     return recommendationsDB.read(userId, ConsistencyLevel.ONE);
// }

// Note: never use eventual consistency for inventory or order placement.

// io.thecodeforge — System Design tutorial

// Decision tree for choosing consistency model

// Is the data critical for correctness? (money, inventory, user intent)
//   YES -> Is the write throughput > 10K ops/sec?
//     YES -> Consider using a strongly consistent store with sharding (e.g., Spanner, CockroachDB)
//     NO -> Use strong consistency (e.g., Raft-based, single leader)
//   NO -> Is the data user-facing and user expects to see their own writes?
//     YES -> Use read-your-writes consistency
//     NO -> Use eventual consistency

// Example: logging service
// Logs are append-only, never updated. Stale reads are acceptable.
// Use eventual consistency with high throughput.

// Example: payment service
// Must not double-charge. Use strong consistency with idempotency keys.

// Example: social media feed
// Users expect to see their own posts immediately (read-your-writes).
// Others can see stale posts (eventual).
// Use session consistency for the writer, eventual for readers.