Cassandra LOCAL_ONE Write Loss — Multi-DC Failure Patterns

Every time you hit 'like' on a social media post, check your Uber driver's live location, or see your Netflix watch history load in under a second — there's a non-trivial chance Cassandra is doing the heavy lifting underneath. It was built by Facebook engineers to handle the inbox search problem: hundreds of millions of writes per day, globally distributed, with zero acceptable downtime. Relational databases buckled under that load. Cassandra didn't.

The core problem Cassandra solves is the tension between scale, availability, and write throughput that kills traditional RDBMS systems. When you need to ingest millions of events per second across datacenters in Tokyo, Frankfurt, and Ohio simultaneously, you can't afford a single master node, a two-phase commit, or a schema that requires costly JOINs. Cassandra abandons all three. It trades the strong consistency guarantee of SQL for eventual consistency and gives you fine-grained control over exactly how eventual that consistency is — per query.

After working through this article, you'll be able to design a Cassandra data model from scratch using partition keys and clustering columns, explain exactly what happens under the hood during a write (Memtable → CommitLog → SSTable), configure replication for a multi-datacenter cluster, tune consistency levels intelligently based on your SLA, and avoid the five production mistakes that turn a fast Cassandra cluster into a slow one.

The Cassandra Data Model: Partition Keys, Clustering Columns, and Physical Storage

Cassandra's data model is deceptively simple: a table is a collection of rows, and each row is identified by a primary key. But that primary key has two parts: the partition key determines which node stores the row, and the clustering columns determine the order within that partition.

Here's the thing most docs gloss over: the partition key isn't just a lookup key. It's the unit of physical locality. All rows sharing the same partition key live on the same node and are stored contiguously on disk. That's why bad partition key design (e.g., using a timestamp as the sole partition key) creates hot spots — all writes for that second hit one node.

Internally, each partition is stored as a single row inside a Memtable, then flushed to an SSTable on disk. When you query by partition key, Cassandra uses an in-memory index (the partition summary) to find which SSTable contains that partition. It'll read only that SSTable, not the whole file. Bloom filters help skip SSTables that definitely don't contain your partition.

The clustering columns sort rows within a partition. This ordering is physical — it's how data is laid out on disk. So if you query with a range on the first clustering column, Cassandra can skip rows efficiently. This is a big deal for time-series workloads where you often query recent data.

-- TheCodeForge — Cassandra data model for a user timeline
CREATE TABLE io.thecodeforge.user_timeline (
    user_id UUID,                         -- partition key: all tweets for one user on one node
    tweet_id TIMEUUID,                    -- clustering column: sorted by time
    tweet_text TEXT,
    created_at TIMESTAMP,
    PRIMARY KEY (user_id, tweet_id)
) WITH CLUSTERING ORDER BY (tweet_id DESC);

-- Query: get the 20 most recent tweets for a user (scans only one partition, uses sorted order)
SELECT * FROM io.thecodeforge.user_timeline
WHERE user_id = 123e4567-e89b-12d3-a456-426614174000
ORDER BY tweet_id DESC
LIMIT 20;

Write Path Internals: Memtable, CommitLog, and SSTable

When you issue a write to Cassandra, the coordinator node sends it to all replicas that should hold the partition (based on replication strategy). Each replica does this:

1. Writes the mutation to the CommitLog (append-only, sequential writes on disk) — this ensures durability even if the node crashes before the Memtable flushes.
2. Applies the mutation to an in-memory structure called the Memtable (a sorted map of partition key → row data).
3. Returns success to the coordinator — as soon as it's in both CommitLog and Memtable.
4. Later, when the Memtable is full (default 128 MB) or a flush is triggered, it's sorted and written to disk as an immutable SSTable. The CommitLog entries for that Memtable are then truncated.

The key insight: writes never touch a disk read path until flush. That's why Cassandra write throughput is so high — it's essentially sequential CommitLog writes plus memory operations. But that speed has a cost: if you lose power between node 3 (return success) and node 4 (flush), the Mutation is replayed from the CommitLog on startup. The CommitLog is your safety net.

SSTables are immutable — once written, never modified. That means deletions and updates don't modify existing data; they write a new tombstone (for deletes) or a newer timestamped version. Old data becomes garbage that must be compacted away. That's why tombstones are dangerous: if you create a delete-heavy workload without frequent compactions, SSTables pile up with dead rows, inflating read latency.

package io.thecodeforge.cassandra;

import com.datastax.oss.driver.api.core.CqlSession;

/**
 * TheCodeForge — Demonstrates write path behavior: consistency level affects durability.
 * Never run this against a production cluster without understanding the impact.
 */
public class WritePathDebug {
    public static void main(String[] args) {
        try (CqlSession session = CqlSession.builder().build()) {
            // This write returns immediately after one replica acknowledges
            // But if that replica crashes before CommitLog flush, the write is lost
            session.execute(
                "INSERT INTO io.thecodeforge.user_timeline (user_id, tweet_id, tweet_text) " +
                "VALUES (123e4567-e89b-12d3-a456-426614174000, now(), 'Hello Cassandra!') " +
                "USING CONSISTENCY ONE;"
            );
            System.out.println("Write acknowledged. But is it durable? Check CommitLog on replica.");
        }
    }
}

Visual: Cassandra Write Path — CommitLog → Memtable → SSTable

The write path is the most critical flow in Cassandra because it determines both throughput and durability. This diagram visualizes the three main stages: the CommitLog (durable, append-only log), the Memtable (in-memory sorted buffer), and the SSTable (immutable disk file). The sequence is: (1) write arrives, (2) immediately written to CommitLog for crash recovery, (3) applied to Memtable for fast reads, (4) acknowledged to client once both are done, (5) later, Memtable is flushed to an SSTable on disk, and the CommitLog segment is truncated.

The architectural insight: writes are fast because step 2 and 3 are sequential I/O and memory operations. The flush (step 5) happens asynchronously in the background. This design prioritizes write throughput over read latency — reads may have to consult multiple SSTables, but writes are nearly as fast as the network and memory allow.

The diagram below uses a mermaid flowchart to capture the flow.

Replication: Strategies, Snitch, and Multi-Datacenter Placement

Cassandra distributes data across nodes using a consistent hash ring. Each node owns a range of token values. A partition key is hashed to a token, and the node that owns that token range is the primary replica. Additional replicas are placed on the next N nodes in the ring (where N = replication factor).

But that's only half the story. The actual placement depends on the replication strategy and the snitch. The snitch tells Cassandra about network topology — which nodes are in which rack and datacenter. Without a proper snitch, Cassandra will place replicas on nodes in the same rack, defeating fault tolerance.

Here's the gotcha: if your snitch is misconfigured (e.g., using SimpleSnitch when you have multiple datacenters), Cassandra will think all nodes are in one datacenter and will place all replicas in that single DC — the other DC gets zero data. You'll have a false sense of redundancy.

# TheCodeForge — cassandra.yaml snitch configuration
endpoint_snitch: GossipingPropertyFileSnitch

# Then in cassandra-rackdc.properties:
# dc=dc_west
# rack=rack1

# Keyspace creation using NetworkTopologyStrategy
CREATE KEYSPACE io.thecodeforge.myapp WITH REPLICATION = {
  'class': 'NetworkTopologyStrategy',
  'dc_west': 3,
  'dc_east': 2
};

# Verify replication with:
# SELECT * FROM system_schema.keyspaces WHERE keyspace_name = 'io.thecodeforge.myapp';

Tunable Consistency: What Each Level Actually Means for Latency and Correctness

Cassandra's consistency levels let you choose how many replicas must respond before a query returns. This is per-query — you can use strong consistency for critical operations and eventual consistency for everything else.

Here's the trade-off: higher consistency = higher latency + lower availability. If you require ALL replicas to agree, a single node failure makes that query impossible. If you require ONE replica, you get fast responses but risk stale reads or lost writes.

The production rule: use LOCAL_QUORUM for writes and reads within the same datacenter. This provides strong consistency within that DC and survives one node failure (RF=3). For global reads that don't need the latest data, use LOCAL_ONE.

A common mistake is using QUORUM in a multi-DC world: QUORUM means majority of all replicas, not majority in each DC. In a 3+3 DC setup, QUORUM requires 4 replicas – which could be 3 in DC1 and 1 in DC2. That cross-DC read adds latency and if DC2 is unreachable, QUORUM may fail. Use LOCAL_QUORUM instead.

-- TheCodeForge — Using consistency levels strategically

-- Critical write: must survive a single node failure
INSERT INTO io.thecodeforge.orders (order_id, amount, status)
VALUES (uuid(), 99.99, 'confirmed')
USING CONSISTENCY LOCAL_QUORUM;

-- Non-critical read: latest may be a few seconds old
SELECT * FROM io.thecodeforge.recommendations WHERE user_id = ?
CONSISTENCY LOCAL_ONE;

-- Batch read: need consistent snapshot across multiple partitions
-- Use SERIAL (paxos) only for lightweight transactions (LWT)
BEGIN BATCH
  UPDATE io.thecodeforge.user_timeline SET is_visible = false
  WHERE user_id = ? AND tweet_id = ?
  IF EXISTS
APPLY BATCH;  -- LWT requires SERIAL consistency

-- Checking effective consistency with drivers:
-- In Java driver: SimpleStatement.withDefaultReadConsistency()

Quick-Reference: Consistency Levels and Their Effects

Choosing the right consistency level per query is the key to balancing performance and correctness. This table consolidates the available levels, their guarantees, and recommended use cases. Unlike the high-level comparison earlier, this table includes the exact number of replicas required and how failure tolerance changes with replication factor.

The rule of thumb: for business-critical operations within a datacenter, use LOCAL_QUORUM. For global operations requiring strong consistency across datacenters, use EACH_QUORUM — but expect 20-100ms latency. For high-throughput background tasks, use LOCAL_ONE or ONE.

Important nuance: LOCAL_SERIAL and SERIAL are used for lightweight transactions (LWTs) like conditional updates. They use Paxos protocol internally and are significantly slower — avoid in hot paths.

-- Check current consistency level in CQLSH
CONSISTENCY;
-- Output: Current consistency level is LOCAL_QUORUM.

-- Set per query
SELECT * FROM io.thecodeforge.orders WHERE order_id = ?
CONSISTENCY LOCAL_ONE;

-- In Java driver:
// SimpleStatement stmt = SimpleStatement.newInstance("SELECT ...");
// stmt = stmt.setExecutionProfile(session.getContext().getExecutionProfile()
//     .withConsistencyLevel(DefaultConsistencyLevel.LOCAL_ONE));

Compaction Strategies: SizeTiered, Leveled, and TimeWindow – When to Use Each

SSTables are immutable, so over time you accumulate many small files. Compaction merges them into larger ones, discards tombstones and overwritten data, and reclaims disk space. Cassandra offers three main strategies:

SizeTieredCompactionStrategy (STCS): merges SSTables of similar size. Default. Simple, but can lead to write amplification (multiple merges of the same data) and space amplification (temporary disk double during compaction). Best for write-heavy, read-rare workloads where uniformity is acceptable.

LeveledCompactionStrategy (LCS): organizes SSTables into levels (L0, L1, L2...). Each level is 10x larger than the previous. Compaction occurs within a level, guaranteeing that 90% of reads hit at most one SSTable per level. Results in lower read latency but higher write amplification (more compaction work for each write). Best for read-heavy workloads, high consistency, or systems where read tail latency matters.

TimeWindowCompactionStrategy (TWCS): designed for time-series data. SSTables are grouped by time window (e.g., one hour). Compaction only merges SSTables within the same window. Old windows are left untouched. Ideal for metrics data where older data is rarely queried and tombstones from TTLs can be efficiently removed per window.

The trick: choose based on your write/read ratio and access pattern. STCS works for most cases, but don't use it with high TTL workloads – it'll keep merging tombstones into ever-larger SSTables, wasting disk and read time. TWCS is far more efficient for TTL-heavy tables.

-- TheCodeForge — Compaction strategy configuration

-- For a read-heavy user profile table:
ALTER TABLE io.thecodeforge.users WITH compaction = {
  'class': 'LeveledCompactionStrategy',
  'sstable_size_in_mb': 160
};

-- For a time-series metrics table with 30-day TTL:
CREATE TABLE io.thecodeforge.metrics (
    sensor_id UUID,
    hour_bucket INT,
    timestamp TIMESTAMP,
    value DOUBLE,
    PRIMARY KEY ((sensor_id, hour_bucket), timestamp)
) WITH compaction = {
  'class': 'TimeWindowCompactionStrategy',
  'compaction_window_unit': 'HOURS',
  'compaction_window_size': 1
} AND default_time_to_live = 2592000; -- 30 days

-- Check current compaction strategy:
SELECT table_name, compaction FROM system_schema.tables WHERE keyspace_name = 'io.thecodeforge';

Compaction Strategy Decision Matrix — Choosing the Right Strategy for Your Workload

Cassandra offers three compaction strategies, each optimized for different access patterns. The decision comes down to your read/write ratio, data lifetime (TTL), and whether you prioritize write throughput or read latency. Below is a decision matrix comparing STCS, LCS, and TWCS across key dimensions.

How to use this matrix: Identify your workload category (column) and find the recommended strategy. Then check the trade-offs in terms of write amplification, read amplification, disk space overhead, and compaction frequency. The 'Best For' column gives a quick verdict.

-- Example: If you have a user_profile table that is read-heavy (high reads per write):
ALTER TABLE io.thecodeforge.user_profile WITH compaction = {
  'class': 'LeveledCompactionStrategy',
  'sstable_size_in_mb': 160
};

-- Example: Time-series sensor data with TTL=7 days:
CREATE TABLE io.thecodeforge.sensor_data (
    sensor_id UUID,
    day_bucket INT,
    ts TIMESTAMP,
    value DOUBLE,
    PRIMARY KEY ((sensor_id, day_bucket), ts)
) WITH compaction = {
  'class': 'TimeWindowCompactionStrategy',
  'compaction_window_unit': 'DAYS',
  'compaction_window_size': 1
} AND default_time_to_live = 604800;

-- Verify current strategy:
SELECT table_name, compaction FROM system_schema.tables WHERE keyspace_name = 'io.thecodeforge';

Tombstones and Bloom Filters — How Deletes Are Handled and How Reads Are Accelerated

Tombstones are Cassandra's mechanism for handling deletes. When you delete a row, Cassandra writes a tombstone — a marker that says 'this data is considered deleted as of timestamp X.' The original data is not removed immediately; it remains in the SSTable until compaction. During a read, Cassandra scans through SSTables and skips any cell with a tombstone whose timestamp is newer than the data. This makes deletes and TTLs (which also create tombstones) deceptively expensive: each tombstone must be read and ignored, adding latency.

Bloom filters are the counterbalance. Before reading an SSTable, Cassandra checks the Bloom filter — a probabilistic data structure that tells if an SSTable definitely does NOT contain the requested partition key. If the filter says 'no,' the SSTable is skipped entirely. This drastically reduces read latency, especially when many SSTables exist. However, Bloom filters have a false positive rate (default 1 in 10,000) — sometimes they say 'maybe' when the data isn't there, causing a wasted read. The filter is stored in memory and is keyed by partition key.

Tombstone gotchas: - A partition with >100K tombstones can cause a read timeout even if the live data is just a few rows. - You can monitor tombstones with nodetool cfstats — look for 'SSTable count' and 'Tombstone count'. - The gc_grace_seconds setting (default 10 days) determines how long a tombstone must exist before compaction can remove it. This is to prevent resurrecting data from other nodes that may not have seen the delete yet. If you're sure all replicas have the delete, you can lower this value to reduce tombstone duration.

Bloom filter tuning: - bloom_filter_fp_chance (default 0.01 = 1% false positive) trades memory for accuracy. Lower values use more memory but reduce wasted reads. - Bloom filters are stored in off-heap memory. If memory is tight, you can increase the false positive chance to 0.1, but expect higher read latency. - Use nodetool cfstats to see 'Bloom filter false positives' — if this is high, consider reducing bloom_filter_fp_chance.

# Check tombstone and bloom filter stats for a table
nodetool cfstats io.thecodeforge.my_table | grep -E 'Bloom|Tombstone|SSTable'

# Sample output:
# Bloom filter false positives: 0
# Bloom filter false ratio: 0.00000
# Bloom filter space used: 1048576
# SSTable count: 12
# Tombstone count: 5432
# Tombstone ratio: 0.10

# To lower bloom filter false positive chance (uses more memory):
# In cassandra.yaml: bloom_filter_fp_chance: 0.001
# Or per table via CQL:
ALTER TABLE io.thecodeforge.my_table WITH bloom_filter_fp_chance = 0.001;

# To force compaction to remove tombstones early (after reducing gc_grace_seconds):
nodetool compact io.thecodeforge my_table