System Design Interview: Fan-out OOM & Duplicate Tweets

System design interviews are the highest-signal round in any senior engineering hiring process. They separate engineers who can implement from engineers who can think at scale. A candidate who writes flawless LeetCode solutions can still completely bomb a system design round because the skills are fundamentally different — one is puzzle-solving, the other is structured ambiguity navigation.

The problem isn't that engineers don't know distributed systems. Most senior candidates have read about consistent hashing, Kafka, and sharding. The problem is they don't know how to structure a 45-minute conversation into a coherent narrative that demonstrates both technical depth and product instinct simultaneously.

By the end of this article you'll have a repeatable, time-boxed framework you can apply to any system design prompt. You'll know exactly what to say in each phase, which trade-offs to surface proactively, how to handle follow-up pressure, and what signals separate a 'strong hire' from a 'hire' at FAANG-level bars.

Phase 1 — Clarify Requirements Before Touching the Whiteboard (Minutes 0–5)

The single biggest mistake candidates make is starting to design immediately. Interviewers deliberately leave the prompt vague because the ability to identify the right questions is itself a senior engineering skill. 'Design Twitter' could mean the timeline, the search, the DM system, or the ad platform — they're completely different problems with completely different bottlenecks.

Spend the first five minutes asking structured clarifying questions in two buckets: functional requirements and non-functional requirements.

Functional requirements define what the system does. Which features are in scope? 'Design Twitter' — are we doing tweet posting, home timeline, follower graph, search, or all of them? Pin down one or two core flows and explicitly park the rest: 'I'll focus on posting tweets and reading the home timeline. I'll note search and DMs as out of scope unless we have time.'

Non-functional requirements define how the system behaves under load. This is where most candidates are too vague. Don't ask 'how many users?' — ask specifically: 'Are we targeting Twitter's actual scale of ~500M daily active users, or is this a startup at 1M DAU?' Then drive the numbers yourself. Assume a write:read ratio, calculate QPS, estimate storage needs. Saying 'let's assume 500M DAU, 100M tweets per day, roughly 1,200 tweet writes per second and maybe 10x that for reads — so about 12,000 timeline reads per second as a floor, likely higher with fan-out' shows the interviewer you can reason from first principles rather than recite memorized numbers.

Also clarify consistency requirements explicitly. Is eventual consistency acceptable for the timeline? Almost certainly yes — a user seeing a tweet 2 seconds late is tolerable. For payments or inventory? Never. Lock in the consistency model you're designing to before you propose any components, because it determines whether you can use async fan-out at all.

// ── PHASE 1: REQUIREMENT CLARIFICATION SCRIPT ──────────────────────────────
// Use this verbatim in your next interview. Adapt the numbers to the prompt.

// STEP 1 — Functional Scope
"Before I start designing, I want to make sure we're aligned on scope.
 Let's say we're building Twitter's core. I'm going to focus on:
   (1) A user posting a tweet
   (2) A user reading their home timeline
 Out of scope for now: search, DMs, ads, trending topics.
 Does that work for you, or would you like me to reprioritize?"

// STEP 2 — Scale Estimation (do this out loud, show your work)
"Let me size this system:
  - 500M daily active users
  - Assume 20% post content: ~100M tweets/day
  - 100M tweets / 86,400 seconds ≈ 1,200 tweet writes/second (peak ~3x = 3,600 wps)
  - Read:Write ratio is typically 100:1 for social feeds
  - So reads: ~120,000 reads/second (peak ~360,000 rps)
  - Tweet storage: avg 200 bytes per tweet * 100M/day = 20GB/day, ~7TB/year
  - Media (images/video) stored separately in object storage — not in the tweet DB"

// STEP 3 — Non-Functional Requirements
"A few more constraints I want to nail down:
  - Latency: home timeline should load in under 200ms p99
  - Availability: 99.99% uptime (four nines) — no scheduled downtime
  - Consistency: eventual consistency is OK for the timeline
    (a user seeing a tweet 2 seconds late is fine)
  - Durability: tweets must never be lost once written — acknowledged writes
    must survive a single node failure
  - Geo: assume global, so we need multi-region support with latency-based routing"

// OUTPUT: You now have a design contract. Everything you build next
// is justified against these numbers. This is what separates architects
// from engineers who just know technology names.

Phase 2 — High-Level Design and the API Contract (Minutes 5–20)

Once requirements are locked, draw the 30,000-foot view before zooming in anywhere. Most candidates zoom in too fast — they start talking about database sharding before they've even established what services exist. Interviewers want to see that you can hold the whole system in your head before optimizing any part of it.

Start with the client-to-server path. Draw: client → load balancer → API gateway → application services → data stores. Then define your API contract for the core flows. This is non-negotiable for senior roles — defining the API before designing the backend proves you think contract-first, which is how production systems are actually built. Teams agree on the API surface first, then build services independently against that contract.

For tweet posting: POST /v1/tweets, body: {user_id, content, media_ids[]}, response: {tweet_id, created_at}. For timeline: GET /v1/users/{user_id}/timeline?cursor={cursor}&limit=20, response: {tweets: [...], next_cursor: string}. Note the cursor-based pagination — explain why explicitly: offset pagination requires the database to scan and discard N rows to reach the offset, which degrades to a full table scan at depth. Cursor pagination hits the index directly.

Decompose into services early. For Twitter, you need at minimum: a Tweet Service (writes and reads), a Timeline Service (fan-out and reads), a User Service (auth and profiles), and a Media Service (upload/serve images). Keep services small and name them after the business capability, not the technology. Don't say 'the Java microservice' — say 'the Timeline Service.' This signals domain-driven thinking and maps directly to how you'd structure teams around the system.

For each service, identify the primary data it owns. The Tweet Service owns a tweets table. The Timeline Service owns precomputed timeline caches. The User Service owns the social graph. Call out the anti-pattern explicitly before the interviewer asks about it: 'I want to be clear that each service owns its own datastore. The Timeline Service does not query the Tweet Service's database directly. Cross-service communication happens via APIs or async events on a message queue. Shared databases create hidden coupling that makes services impossible to scale or deploy independently — I've seen this pattern cause production outages where a slow query from one service brought down an unrelated service sharing the same DB connection pool.'

// ── PHASE 2: HIGH-LEVEL DESIGN WALKTHROUGH ──────────────────────────────────
// Narrate this while drawing on the whiteboard. Never draw in silence.

// STEP 1 — API Contract (define before backend)

API DESIGN — Tweet Posting Flow:
  POST /v1/tweets
  Request:  { user_id: UUID, content: string(280), media_ids: UUID[] }
  Response: { tweet_id: UUID, created_at: ISO8601, status: 'published' }

API DESIGN — Home Timeline Read:
  GET /v1/users/{user_id}/timeline?cursor={opaque_cursor}&limit=20
  Response: {
    tweets: [ { tweet_id, author_id, content, media_url, created_at, like_count } ],
    next_cursor: "eyJ0d2VldF9pZCI6IjEyMzQ1NiJ9"  // base64-encoded last tweet ID
  }

  // Why cursor pagination on the timeline read?
  // Offset-based: SELECT * FROM tweets LIMIT 20 OFFSET 10000
  //   → DB scans 10,020 rows and discards 10,000. O(n) cost. Breaks at scale.
  // Cursor-based: SELECT * FROM tweets WHERE id < :last_seen_id LIMIT 20
  //   → Uses the index directly. O(log n). Stable even when rows are inserted.

// STEP 2 — Service Decomposition

  [Mobile / Web Client]
       ↓ HTTPS
  [CDN — static assets, cached timeline responses for inactive users]
       ↓ cache miss
  [API Gateway — rate limiting, auth token validation, routing]
       ↓
  ┌──────────────────────────────────────────────┐
  │  Tweet Service    Timeline Service           │
  │  User Service     Notification Service       │
  │  Media Service    Search Service (out-scope) │
  └──────────────────────────────────────────────┘
       ↓                    ↓
  [Tweet DB]         [Timeline Cache — Redis]
  [User DB]          [Fan-out Queue — Kafka]
  [Object Store]     [Notification Queue — Kafka]

// STEP 3 — Call out the coupling anti-pattern proactively
"One thing I want to be explicit about: each service owns its own datastore.
 The Timeline Service does NOT query the Tweet Service's database directly.
 Communication between services is via APIs or async events on a message queue.
 Shared databases create hidden coupling — I've seen this cause incidents where
 a slow analytics query from the reporting service saturated the connection pool
 on the tweet DB and caused write timeouts on the hot path. Separate datastores
 eliminate that failure mode entirely."

Phase 3 — Deep Dive on the Hard Problems (Minutes 20–40)

This is where the interview is won or lost. After the high-level design, a good interviewer will steer you toward the hardest sub-problem in your design. For Twitter, that's the fan-out problem: when Katy Perry tweets, how do you push that tweet to 150M followers without your system catching fire?

There are two approaches — fan-out on write (push) and fan-out on read (pull) — and neither is universally correct. This is a classic system design trade-off that interviewers use specifically because it has no single right answer. The right choice depends on the read:write ratio and the distribution of follower counts in your user base.

Fan-out on write: when a tweet is posted, the Tweet Service publishes an event to a fan-out queue. A fleet of fan-out workers picks up the event and writes the tweet_id into each follower's precomputed timeline in Redis. Timeline reads are then a simple Redis ZSET lookup — sub-millisecond. The cost? Writing one tweet for a celebrity with 50M followers means 50M Redis writes. This is write amplification at its most extreme. At 3,600 writes per second at peak, and assuming the average user has 1,000 followers, that's 3.6M fan-out writes per second for average users alone — before you account for any celebrity traffic.

Fan-out on read: when a user opens their timeline, the Timeline Service queries the social graph to get their followed accounts, fetches recent tweets from each of those accounts, merges and sorts them, and returns the result. No write amplification — but read time scales linearly with the number of followed accounts. Following 5,000 accounts means up to 5,000 DB lookups on every timeline refresh, which violates the 200ms p99 SLA we set in Phase 1.

The production answer, used by Twitter and Instagram, is a hybrid: fan-out on write for normal users (fewer than 10K followers), fan-out on read for celebrities (more than 10K followers). When Katy Perry's tweet appears in your timeline, it was lazily fetched at read time and merged in memory with your precomputed timeline from non-celebrity follows. This caps write amplification at a manageable level while keeping reads fast for the overwhelming majority of cases. The 10K threshold is a tunable configuration value, not a magic number — you'd calibrate it against your actual follower distribution and Redis write capacity.

// ── PHASE 3: FAN-OUT DEEP DIVE — THE HARD PROBLEM ──────────────────────────

// HYBRID FAN-OUT ALGORITHM (production-grade)

When a tweet is posted:
  1. Tweet Service writes tweet to tweets DB (source of truth)
  2. Tweet Service publishes event to Fan-out Queue (Kafka topic: 'new-tweets')

Fan-out Worker picks up the event:
  3. Fetch author's follower list from User Service (graph DB or followers table)
  4. Check follower count:
     IF author.follower_count < CELEBRITY_THRESHOLD (e.g., 10,000):
       → PUSH path: for each follower_id, write tweet_id to their Redis timeline
           ZADD timeline:{follower_id} {timestamp_score} {tweet_id}
           // Redis ZSET keeps timeline sorted by time automatically
           // Trim to last 800 entries to bound memory per user:
           ZREMRANGEBYRANK timeline:{follower_id} 0 -801
     ELSE:
       → PULL path: mark tweet as 'celebrity-tweet', store only in tweets DB
           No fan-out write. Will be fetched at read time.
           SET celebrity_tweet:{tweet_id} 1 EX 86400  // 24hr marker

When a user requests their timeline:
  5. Timeline Service reads precomputed timeline from Redis:
       ZREVRANGE timeline:{user_id} 0 19  // last 20 tweet_ids, newest first
  6. Identify which users the requester follows AND have celebrity status:
       celebrity_follows = graph.get_celebrity_follows(user_id)
  7. For each celebrity, fetch their latest tweets from tweets DB:
       SELECT tweet_id FROM tweets
       WHERE author_id IN (:celebrity_ids)  // batch query, not N+1
       ORDER BY created_at DESC LIMIT 20 PER author
  8. MERGE precomputed timeline + celebrity tweets in memory:
       merged = sort_by_time(redis_tweets + celebrity_tweets)[:20]
  9. Hydrate tweet_ids → full tweet objects via Tweet Service (batch GET)
  10. Return hydrated, merged, paginated timeline to client

// WHY THIS WORKS:
// - Normal users (99.9% of authors): write amplification is bounded
//   1,000 followers * 3,600 writes/sec = 3.6M Redis ops/sec — achievable
//   with a Redis cluster at this scale
// - Celebrities: no write amplification. 50M followers * 0 writes = 0
// - Read path: one Redis lookup + a handful of DB queries for celebrities
//   This keeps p99 latency under 200ms even at peak scale

// EDGE CASES TO CALL OUT IN YOUR INTERVIEW:
// 1. Threshold transition: if @JohnDoe gains 10,001 followers, do we backfill
//    existing timelines? No — switch the threshold prospectively.
//    The slight inconsistency window is acceptable under eventual consistency.
// 2. Redis eviction: if a user hasn't logged in for 30 days, their timeline
//    key expires. On return, trigger an async cold-start timeline rebuild.
//    The user sees a loading state for ~500ms on first login, then normal UX.
// 3. Follower list pagination: fetching 50M follower IDs in one call will OOM
//    your fan-out worker. Stream in batches of 10,000 using cursor pagination.
// 4. At-least-once delivery: Kafka guarantees at-least-once. A fan-out worker
//    crash mid-batch could re-process the event. Make ZADD idempotent —
//    re-adding the same member with the same score is a no-op in Redis.
//    Normalize timestamp scores to second precision before writing.

Phase 4 — Trade-offs, Bottlenecks, and How to Close Strong (Minutes 40–45)

The last five minutes are your chance to demonstrate that you think about systems holistically — not just 'does it work?' but 'how does it fail, and how do we recover?' Strong candidates proactively surface the weaknesses in their own design before the interviewer has to find them. This is not just a performance trick — it reflects genuine production experience, because engineers who've actually operated systems at scale know that the interesting questions are always about the failure modes, not the happy path.

Walk through your design and call out at least three potential bottlenecks with mitigations. Don't wait to be asked. For our Twitter design: (1) the fan-out queue — if Kafka falls behind during a traffic spike, timeline freshness degrades. Mitigation: monitor consumer lag per partition, auto-scale fan-out workers on lag metric, implement a dead-letter queue for failed fan-out events, and fall back to full fan-out-on-read as a degraded mode if lag exceeds five minutes. (2) Redis memory — precomputing timelines for 500M users at ~800 tweet_ids each, stored as 8-byte integers, is roughly 3.2TB of Redis storage. Manageable with a Redis cluster, but requires LRU eviction, timeline key expiry for inactive users, and strict enforcement that only tweet_ids (not full objects) are stored in Redis. (3) The single-region failure mode — the design as described has no geographic failover. For 99.99% global availability, deploy identical stacks in three regions with latency-based DNS routing, and replicate tweets asynchronously across regions via Kafka MirrorMaker 2, accepting ~500ms of cross-region replication lag.

Then close with a design summary — this is almost universally skipped by candidates but it's one of the most powerful things you can do: 'To summarize — we designed a Twitter clone handling 1,200 tweet writes and 120,000 timeline reads per second. The key architectural insight was the hybrid fan-out strategy that caps write amplification while keeping read latency under 200ms p99. The main trade-off was operational complexity — the fan-out worker requires careful idempotency handling and batch checkpointing that would not exist in a simpler pull-only design.' One paragraph. Architecture recapped. Trade-off named. This demonstrates that you can communicate to a non-technical stakeholder or an engineering manager, not just to the person next to you at a whiteboard.

Finally, leave time to ask the interviewer a genuine question about their actual systems. 'How does your team handle the fan-out problem today — did you go hybrid, or take a different approach?' This signals collaborative instinct and intellectual curiosity. Interviewers remember candidates who made the conversation feel like a peer discussion rather than an oral exam.

// ── PHASE 4: BOTTLENECK REVIEW AND CLOSING STATEMENT ────────────────────────

// PROACTIVE BOTTLENECK IDENTIFICATION (say this before they ask)

"Let me stress-test my own design before we wrap up."

// BOTTLENECK 1 — Fan-out Queue Lag
"If we get a sudden traffic spike — say, a major world event drives 10x normal
 tweet volume — our Kafka consumer group (fan-out workers) may fall behind.
 The symptom: users see stale timelines. The mitigation:
  - Monitor consumer lag per Kafka partition (alert if lag > 30 seconds)
  - Auto-scale fan-out worker pods based on lag metric (HPA in Kubernetes)
  - For extreme outliers, shed load: if fan-out lag exceeds 5 minutes,
    temporarily switch all users to fan-out-on-read as a degraded mode
  - Dead-letter queue for fan-out events that fail after 3 retries"

// BOTTLENECK 2 — Redis Memory at Scale
"500M users * 800 tweet_ids * 8 bytes each = ~3.2TB of Redis storage.
 Achievable with a Redis cluster, but requires active management:
  - Expire timeline keys for users inactive > 30 days (EXPIRE command)
  - Store only tweet_id (int64, 8 bytes) not full tweet objects in Redis
  - Use Redis cluster with consistent hashing across 10+ shards
  - Monitor memory fragmentation ratio — above 1.5 means Redis is
    allocating but not efficiently reusing memory; trigger MEMORY PURGE
  - Set a hard cap of 800 entries per timeline ZSET via ZREMRANGEBYRANK"

// BOTTLENECK 3 — Single-Region Failure
"My design so far is single-region. For 99.99% availability globally:
  - Deploy identical stacks in 3 regions: US-East, EU-West, AP-Southeast
  - Use latency-based DNS routing to pin users to their nearest region
  - Tweets written to local region, asynchronously replicated to other
    regions via Kafka MirrorMaker 2 with ~500ms replication lag
  - Conflict resolution: last-write-wins with tweet_id as tiebreaker
  - Timeline reads always served from local region Redis — cross-region
    fan-out happens async and is acceptable under eventual consistency"

// CLOSING SUMMARY (always do this — most candidates skip it)
"To summarize what we built:
  - Core entities: tweets, users, follower graph, precomputed timelines
  - Write path: Tweet Service → Kafka → Fan-out Workers → Redis + Tweet DB
  - Read path: API Gateway → Timeline Service → Redis ZSET + celebrity DB queries
  - Key trade-off: hybrid fan-out adds operational complexity — idempotency
    handling and batch checkpointing — but is the only approach that keeps
    both write amplification and read latency bounded simultaneously
  - Scale targets: 1,200 wps, 120,000 rps, 7TB/year tweet storage, 3.2TB Redis
  - Reliability target: 99.99% via multi-region active-active, async replication"

// YOUR QUESTION TO THE INTERVIEWER
"One thing I'd genuinely love to know — how does your team actually handle
 the fan-out problem today? Did you go hybrid, or take a different approach?
 And what was the hardest operational problem you ran into with it?"