Design WhatsApp — Exactly-Once Across Server Boundaries

WhatsApp handles over 100 billion messages per day across 2 billion active users. At that scale, the difference between a good design and a great one isn't features — it's whether your system stays alive on a Tuesday afternoon when half of India gets a holiday and everyone texts at once. This isn't a toy problem. It's one of the most sophisticated real-time distributed systems ever built, and interviewers use it precisely because it exposes every weakness in your distributed systems thinking.

The core problem WhatsApp solves is deceptively simple: two people want to exchange text (and now media) in real time, with guarantees around delivery and ordering, without either party having to stay permanently connected to the same server. Underneath that simplicity lurks a minefield: presence detection, message fan-out in group chats, offline message queuing, idempotent delivery, end-to-end encryption key exchange, and media deduplication across petabytes of storage.

By the end of this article you'll be able to walk into a system design interview and sketch the full WhatsApp architecture — connection layer, message routing, storage schema, delivery receipts, group messaging fan-out, media pipeline, and E2E encryption flow — and, more importantly, defend every single choice with concrete trade-offs. Let's build it.

Core Messaging Architecture: WebSocket Connection Management

WhatsApp's real-time communication relies on persistent WebSocket connections. Each user maintains a long-lived TCP connection to one of thousands of chat server nodes. The connection lifecycle is: client connects → gateway routes to least-loaded chat server → chat server assigns a session ID → server registers the user's presence in an in-memory distributed hash table (DHT) backed by Redis.

The chat server is responsible for receiving messages from the client, validating them, persisting to Cassandra, and forwarding to the recipient's chat server. The recipient's chat server then pushes the message over its WebSocket if the recipient is online. If offline, the message is queued in Cassandra with a TTL of 30 days.

Why not use HTTP long-polling? WebSockets reduce per-message overhead from ~1KB (HTTP headers) to ~150 bytes (WebSocket frame). At 100B messages/day, that saves ~85TB of bandwidth daily. Also, WebSockets enable push-based delivery without polling, keeping median latency under 200ms.

package io.thecodeforge.chat;

import io.thecodeforge.chat.model.Session;
import io.thecodeforge.chat.model.UserId;
import io.thecodeforge.storage.RedisClient;

public class WebSocketSessionManager {
    private final RedisClient<String, Session> sessionStore;
    private static final long SESSION_TTL_SECONDS = 3600;

    public Session createSession(UserId userId, String serverId) {
        Session session = new Session(userId, serverId, System.currentTimeMillis());
        String key = "session:" + userId.value();
        sessionStore.setex(key, SESSION_TTL_SECONDS, session);
        return session;
    }

    public Session refreshSession(UserId userId) {
        String key = "session:" + userId.value();
        return sessionStore.get(key)
            .map(s -> { s.refresh(); return s; })
            .orElseThrow(() -> new RuntimeException("Session expired"));
    }

    public void markOffline(UserId userId) {
        sessionStore.del("session:" + userId.value());
        presenceStore.del("presence:" + userId.value());
    }
}

Message Storage and Delivery Semantics

WhatsApp stores every message permanently on the sender's device and temporarily on servers (30 days for delivery, then deleted). The server-side storage schema is designed for high write throughput and point lookups by conversation. The primary data store is Apache Cassandra, chosen for its ability to handle massive write loads with no single point of failure.

Each message has a globally unique ID: (sender_id: long, timestamp_nanos: long, node_id: int). This makes it trivial to deduplicate. Messages are stored in a table keyed by conversation_id (composite of sender and recipient, sorted lexicographically). The table is partitioned by conversation_id, clustered by message_time, so retrieving the last 50 messages is a single partition scan.

Delivery semantics: WhatsApp offers 'sent', 'delivered', and 'read' receipts. The 'delivered' receipt is generated by the recipient's chat server when it pushes the message to the client WebSocket. The 'read' receipt is generated by the client when the user opens the chat. Read receipts require careful ordering: if a message is read before an earlier one is delivered (possible in group chats), the system must preserve causal order.

WhatsApp uses a logical clock per conversation to order messages. Each server increments a local counter and appends it to the conversation's timestamp. Conflicts are resolved by lexicographic ordering of (counter, server_id).

CREATE TABLE IF NOT EXISTS io.thecodeforge.messages_by_conversation (
    conversation_id text,
    message_time timestamp,
    message_id uuid,
    sender_id bigint,
    recipient_id bigint,
    content_blob blob,
    content_type text,
    delivery_status text,
    read_timestamp timestamp,
    PRIMARY KEY ((conversation_id), message_time, message_id)
) WITH CLUSTERING ORDER BY (message_time DESC, message_id ASC)
   AND default_time_to_live = 2592000; -- 30 days

Group Chat Fan-out: Write Fan-out vs Read Fan-out

Group chat is the hardest part of WhatsApp's architecture. A group with 100k members can't afford to write each message 100k times immediately. WhatsApp uses a hybrid approach: - For groups up to 256 members: write fan-out. The sender's server writes the message to Cassandra once, then fans out notifications to all group members' servers. Each member's server fetches the message when the member goes online. - For groups >256 members: read fan-out. The message is written to a 'group feed' row in Cassandra. Members periodically poll (or get push notified) their group feed and fetch new messages since their last seen timestamp. This reduces write amplification but increases read latency for large groups.

Why 256? It's a sweet spot derived from the average group size on WhatsApp (~50) and the cost-benefit analysis of write amplification vs read latency. At 256, write fan-out generates 256 write notification calls, each taking ~10ms, total ~2.5s server time. Read fan-out would require 256 sequential reads from Cassandra (since each member has a different last_seen), also ~2.5s. The crossover point is around 200-300 members, so 256 is a clean power of 2.

WhatsApp also uses a 'group member list cache' in Redis with a per-group version number. When a member joins or leaves, the version increments, and servers invalidate their local cache. This ensures membership changes propagate within seconds.

package io.thecodeforge.chat;

public class GroupFanoutDecider {
    private static final int WRITE_FANOUT_THRESHOLD = 256;

    public FanoutStrategy decide(int groupSize) {
        if (groupSize <= WRITE_FANOUT_THRESHOLD) {
            return FanoutStrategy.WRITE_FANOUT;
        }
        return FanoutStrategy.READ_FANOUT;
    }

    public enum FanoutStrategy {
        WRITE_FANOUT,
        READ_FANOUT
    }
}

Media Storage and Delivery Pipeline

WhatsApp processes 4.5 billion photos and 1 billion videos daily. Media is too large to route through chat servers — that would saturate the internal network. Instead, WhatsApp uses a dedicated media pipeline: 1. Sender uploads the media to a CDN (Akamai) via a pre-signed URL obtained from the chat server. 2. The chat server returns a hash (SHA-256) of the media content, which is used as the media ID. 3. The sender sends a message containing the media ID (not the media itself) to the recipient. 4. The recipient fetches the media from the CDN using the media ID, with the CDN verifying the hash to prevent content substitution.

This design avoids storing media on the chat servers. The CDN handles blob storage, caching, and bandwidth. WhatsApp also uses content-addressable storage: if two users send the same media (e.g., forwarded meme), the CDN stores only one copy. Deduplication saves ~60% storage.

For videos, the upload pipeline includes a transcoding step (run on a Kubernetes job triggered by the upload success event). Transcoding reduces file size by 40-70% and ensures compatibility. The transcoded versions are stored with the same media ID but different quality suffix (480p, 720p, etc.).

{
  "media_id": "sha256:f4a5b1c...",
  "upload_url": "https://cdn.whatsapp.net/upload/...",
  "expires_at": 1712345678,
  "thumbnail_hash": "sha256:abcd123..."
}

End-to-End Encryption (E2E) Implementation

WhatsApp uses the Signal Protocol for E2E encryption, which provides forward secrecy and deniability. The key exchange is based on a pre-key bundle system:

• Each user generates a long-term identity key (IK) and a set of pre-keys (each is a one-time use key pair).
• The user's device uploads its identity key public half and a signed pre-key bundle to the server.
• When Alice wants to message Bob for the first time, she fetches Bob's pre-key bundle from the server, encrypts an initial message with a session key derived from the pre-key, and sends it.
• Bob decrypts with his stored private pre-key, and they establish a symmetric session key for subsequent messages.

This design means the server never has access to the message content. The server only sees the encrypted payload and metadata (sender, recipient, timestamp). The encryption is done client-side.

For group messages, WhatsApp uses a 'sender key' approach: the sender generates a symmetric key and distributes it to all group members using their individual E2E sessions. All members share the same group encryption key, which is rotated when a member leaves.

Key management is the hardest part — if a user loses a private key (e.g., phone reset), all messages encrypted with that key become undecryptable. WhatsApp stores encrypted backups on the server (with the user's 64-digit backup key) and allows restore only from the user's device.

{
  "identity_key_public": "base64encode...",
  "signed_pre_key": {
    "key": "base64encode...",
    "signature": "base64encode..."
  },
  "one_time_pre_keys": [
    {"id": 1, "key": "base64..."},
    {"id": 2, "key": "base64..."}
  ]
}

Scaling Infrastructure: Chat Servers, Presence, and Multi-Region

WhatsApp's chat servers are stateless application servers that hold WebSocket connections and route messages. They sit behind a global load balancer (anycast DNS + L7 proxy). Each user is assigned to a 'home cluster' based on their phone number hash to maintain sticky sessions and reduce cross-region traffic.

Presence (online/offline/last seen) is stored in a separate low-latency key-value store (custom in-memory DHT with Redis persistence). Updates are propagated via a gossip protocol to all clusters within 5 seconds. To reduce update floods (users frequently connect/disconnect on mobile), WhatsApp batches presence updates: if a user toggles online/offline within 60 seconds, only the final state is published.

Multi-region replication: Messages written to Cassandra in the home cluster are replicated asynchronously to a backup region using cross-datacenter replication. If the home cluster fails, traffic is routed to the backup region. The routing change happens within 10 seconds via a global DNS update (weighted records).

Each region has multiple availability zones. Cassandra's rack-aware replication ensures that each message is stored in at least two AZs within the region. The replication factor is 3, with write consistency set to ONE (for write performance) and read consistency to LOCAL_QUORUM to guarantee no stale reads on crash recovery.

global_routing:
  anycast: yes
  load_balancer:
    type: envoy
    zones:
      - us-east-1
      - eu-west-1
      - ap-southeast-1
  home_cluster_assignment:
    hash: phone_number (mod 64)
  fallback:
    if home_cluster_down:
      route_to: nearest_healthy_cluster
      cool_down: 10s