Design Instagram — Hybrid Push-Pull Feed at 100M Followers

Instagram serves over two billion monthly active users, processes roughly 100 million photo and video uploads every day, and is expected to return a personalised feed in under 300 milliseconds. When an interviewer asks you to design it, they're not looking for a diagram of boxes connected by arrows — they're watching whether you can reason about trade-offs at scale, make deliberate architectural decisions, and defend them under pressure. This is one of the most common system design questions in FAANG-level loops, and candidates who haven't internalised the nuances consistently get stuck on the feed generation problem or grossly underestimate storage requirements.

The core challenge Instagram solves is deceptively simple on the surface: store media, show it to followers, let people discover new content. Underneath, it's a collision of three genuinely hard distributed-systems problems — write-heavy media ingestion, read-heavy personalised feed delivery, and near-real-time social graph queries — all happening concurrently at planetary scale. Each of those problems demands different storage engines, caching strategies, and consistency models, and they have to coexist inside one coherent product.

By the end of this article you'll have a complete, defensible Instagram design you can present in a 45-minute interview. You'll know the exact numbers to anchor your estimates, the right database choices for each data type, how to generate feeds without melting your servers, where CDNs fit, how to shard your data, and — critically — which trade-offs to call out proactively so the interviewer knows you're thinking like an engineer who has shipped things to production, not just read about it.

Why the Instagram Interview Tests Feed Design

The Design Instagram interview asks you to architect a social feed system for 100M+ followers. The core mechanic is the hybrid push-pull model: pre-compute feeds for high-value users (push) and generate on-read for long-tail users (pull). This balances write amplification against read latency.

In practice, you must decide the fanout threshold — typically around 10K followers. Below that, push to all followers' inboxes. Above that, pull from a timeline cache on read. The key properties: push gives O(1) read but O(followers) write; pull gives O(1) write but O(followers) read. Hybrid keeps both under control.

You use this pattern when writes must be fast (posting a photo) and reads must be fresh (scrolling feed). It matters because naive push at 100M followers means a single post triggers 100M writes — impossible. Hybrid is the only way to serve a billion daily active users without collapsing the database.

Core Component Architecture: Handling 100M Uploads Daily

Designing Instagram isn't just about 'uploading a file.' It's about a decoupled architecture where the Write Path (Uploading) and the Read Path (Feed Generation) are optimized independently. On the write side, we use a Load Balancer to distribute traffic to an API Gateway, which handles authentication and rate limiting. The media itself (Photos/Videos) never touches our relational database; instead, it's streamed to an Object Store like AWS S3 or Google Cloud Storage.

We store the metadata (User ID, Photo URL, Timestamp, Location) in a distributed NoSQL database like Cassandra or a sharded PostgreSQL cluster. This separation ensures that even if our metadata database is busy, our media storage remains performant and durable. To make the images load instantly worldwide, we push them to Edge Locations via a Content Delivery Network (CDN).

package io.thecodeforge.instagram;

import java.util.UUID;
import java.time.Instant;

/**
 * Represents the Metadata record for a high-scale media upload.
 * In production, this would be persisted to a sharded DB cluster.
 */
public class PhotoMetadata {
    private final String photoId;
    private final String userId;
    private final String s3Url;
    private final long timestamp;

    public PhotoMetadata(String userId, String s3Url) {
        this.photoId = UUID.randomUUID().toString();
        this.userId = userId;
        this.s3Url = s3Url;
        this.timestamp = Instant.now().getEpochSecond();
    }

    public void saveToDatabase() {
        // High-level logic for sharded database insertion
        System.out.println("Persisting metadata to shard based on userId: " + userId);
        System.out.println("Photo ID: " + photoId + " | Storage Path: " + s3Url);
    }

    public static void main(String[] args) {
        PhotoMetadata upload = new PhotoMetadata("user_8821", "https://s3.thecodeforge.io/bucket/img_99.jpg");
        upload.saveToDatabase();
    }
}

Database Sharding & Scalability Strategies

A single database instance will fail at Instagram's scale. We must shard our data. The best strategy is to shard by User_ID. This ensures that all photos from a single user live on the same shard, making the 'View Profile' query extremely fast. However, for the 'Global Feed', we might need a secondary index or a specialized search service like Elasticsearch.

We also implement a multi-layered caching strategy: Redis for the 'Latest Feed' (Pre-computed), and an LRU cache at the application level for frequently accessed user profiles. This reduces the DB read pressure by over 80%.

-- io.thecodeforge.instagram - Database Schema Concepts
-- Sharding Key: user_id

CREATE TABLE io_thecodeforge.users (
    user_id BIGINT PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

CREATE TABLE io_thecodeforge.photos (
    photo_id BIGINT PRIMARY KEY,
    user_id BIGINT REFERENCES io_thecodeforge.users(user_id),
    image_path VARCHAR(255) NOT NULL,
    caption TEXT,
    latitude DECIMAL(9,6),
    longitude DECIMAL(9,6),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

-- Index for fast Feed Generation (Sorted by Time)
CREATE INDEX idx_user_photos_time ON io_thecodeforge.photos(user_id, created_at DESC);

Feed Generation: The Push-Pull Hybrid Model

The feed is the heart of Instagram. It must show recent posts from followed users in reverse chronological order (or ranked by engagement). Two classic approaches exist: pull (fan-out on read) and push (fan-out on write). Pull means when a user opens the app, we query all followed users' recent posts and merge. Push means when a user posts, we insert that post into every follower's precomputed feed list.

Pull is efficient for celebrities because you don't push to millions; but it's slow for users following hundreds of accounts because you need many queries. Push is fast for reading but costly for writes, especially for popular users. The solution: use push for regular users (fan-out on write) and pull for users with >1M followers (fan-out on read). This hybrid model balances the trade-offs.

package io.thecodeforge.instagram;

import java.util.*;
import java.util.concurrent.*;

public class FeedGenerationService {
    private static final long CELEBRITY_THRESHOLD = 1_000_000;
    private final FollowerService followerService;
    private final FeedCache feedCache;
    private final ExecutorService fanoutExecutor = Executors.newFixedThreadPool(20);

    public void onNewPost(Post post, String userId) {
        long followerCount = followerService.getFollowerCount(userId);
        if (followerCount < CELEBRITY_THRESHOLD) {
            fanoutToAllFollowers(post, userId);
        } else {
            // celebrity: store post for pull-based retrieval
            feedCache.storeCelebrityPost(userId, post);
        }
    }

    private void fanoutToAllFollowers(Post post, String userId) {
        List<String> followers = followerService.getFollowers(userId);
        for (String followerId : followers) {
            fanoutExecutor.submit(() -> feedCache.addToFeed(followerId, post));
        }
    }

    public List<Post> getFeed(String userId, int limit) {
        // merge cached feed with recent posts from followed celebrities
        List<Post> cachedPosts = feedCache.getFeed(userId);
        List<String> followedCelebrities = followerService.getFollowedCelebrities(userId);
        List<Post> celebrityPosts = feedCache.getRecentCelebrityPosts(followedCelebrities);
        List<Post> merged = mergeAndSort(cachedPosts, celebrityPosts);
        return merged.subList(0, Math.min(limit, merged.size()));
    }

    private List<Post> mergeAndSort(List<Post> a, List<Post> b) {
        // merge two sorted (by timestamp descending) lists
        List<Post> result = new ArrayList<>();
        int i = 0, j = 0;
        while (i < a.size() && j < b.size()) {
            result.add(a.get(i).getTimestamp() >= b.get(j).getTimestamp() ? a.get(i++) : b.get(j++));
        }
        while (i < a.size()) result.add(a.get(i++));
        while (j < b.size()) result.add(b.get(j++));
        return result;
    }
}

Storage Strategy: Object Store, CDN, and Cold Archival

Media storage at Instagram's scale requires a tiered approach. The primary storage is an object store (AWS S3, Google Cloud Storage) because they offer near-infinite capacity, strong durability (99.999999999%), and pay-per-GB pricing. However, serving every image directly from S3 would be too slow for users far from the data center and expensive in egress costs. That's where CDNs come in: we push popular media to edge servers worldwide so users download from a nearby node.

For cold data – photos older than 30 days with zero views – we move them to a cheaper archival tier (Amazon S3 Glacier, Google Cloud Archive) and serve a placeholder if accessed. The CDN cache also holds a copy for a shorter TTL (e.g., 7 days for popular content, 1 day for normal). Videos are stored as HLS segments for adaptive bitrate streaming.

package io.thecodeforge.instagram;

import java.time.*;

public class StorageStrategy {
    enum StorageTier { HOT, COLD, ARCHIVE }

    public StorageTier classifyMedia(String mediaId, Instant lastAccessTime) {
        long daysSinceLastAccess = Duration.between(lastAccessTime, Instant.now()).toDays();
        if (daysSinceLastAccess <= 30) {
            return StorageTier.HOT;
        } else if (daysSinceLastAccess <= 365) {
            return StorageTier.COLD;
        } else {
            return StorageTier.ARCHIVE;
        }
    }

    // Production: this method would invoke S3 lifecycle policies
    public String getMediaUrl(String mediaId, StorageTier tier) {
        switch(tier) {
            case HOT:
                return "https://s3.us-east-1.amazonaws.com/instagram-hot/" + mediaId;
            case COLD:
                return "https://s3.us-west-2.amazonaws.com/instagram-cold/" + mediaId;
            case ARCHIVE:
                return "https://glacier.amazonaws.com/instagram-archive/" + mediaId;
            default:
                throw new IllegalArgumentException("Unknown tier");
        }
    }

    public static void main(String[] args) {
        StorageStrategy s = new StorageStrategy();
        StorageTier tier = s.classifyMedia("photo_abc123", Instant.now().minus(60, ChronoUnit.DAYS));
        System.out.println("Media URL: " + s.getMediaUrl("photo_abc123", tier));
    }
}

Caching Strategy: Multi-Level Cache for Sub-300ms Feeds

To achieve sub-300ms feed loads, we need a multi-layer cache. The first layer is a CDN for static media (images, video thumbs). The second layer is an in-memory cache (Redis) for precomputed feeds of active users. The third layer is an application-level LRU cache for frequently accessed metadata (user profiles, popular photos).

For feed data: we store the top N recent posts for each user in Redis (capped at 1000 per user). When a user posts, we push to followers' feed caches (for non-celebrities) or store in a 'celebrity post list' in Redis. For membership services (follower counts, liked), we use a separate Redis cluster with eventual consistency. Cache invalidation is handled via version numbers: each user has a feed version; when a new version is available (due to new post), the client refetches.

package io.thecodeforge.instagram;

import redis.clients.jedis.Jedis;
import java.util.*;

public class CacheStrategy {
    private static final int FEED_CACHE_SIZE = 1000;
    private final Jedis jedis;

    public CacheStrategy(String redisHost) {
        this.jedis = new Jedis(redisHost);
    }

    public void addPostToUserFeed(String userId, Post post) {
        String key = "feed:" + userId;
        // Use sorted set with timestamp as score
        jedis.zadd(key, post.getTimestamp(), post.getId());
        // Trim to keep only top 1000
        jedis.zremrangeByRank(key, 0, -(FEED_CACHE_SIZE + 1));
    }

    public List<String> getUserFeed(String userId, int start, int count) {
        String key = "feed:" + userId;
        Set<String> ids = jedis.zrevrange(key, start, start + count - 1);
        return new ArrayList<>(ids);
    }

    public long getFeedSize(String userId) {
        return jedis.zcard("feed:" + userId);
    }

    // Cache aside pattern for user profile
    public UserProfile getUserProfile(String userId) {
        String key = "profile:" + userId;
        UserProfile profile = (UserProfile) jedis.get(key); // assuming serialization
        if (profile == null) {
            profile = loadFromDatabase(userId);
            jedis.setex(key, 300, profile.toString()); // 5 min TTL
        }
        return profile;
    }

    private UserProfile loadFromDatabase(String userId) {
        // Placeholder – in reality query sharded DB
        return new UserProfile(userId, "user_" + userId, System.currentTimeMillis());
    }

    public static void main(String[] args) {
        CacheStrategy cache = new CacheStrategy("localhost:6379");
        cache.addPostToUserFeed("user123", new Post("photo1", 1000000L));
        System.out.println("Feed: " + cache.getUserFeed("user123", 0, 10));
    }
}

Capacity Estimation: The Math They Actually Ask For, But Nobody Does Right

Everyone waves hands about "millions of users" in interviews. But when a senior engineer asks you how much storage you need per day, or what bandwidth your CDN has to provision, they're not being pedantic — they're testing if you've dealt with real infra limits. Let's do the numbers, because a 50TB storage requirement you didn't anticipate will kill your architecture faster than any cache miss.

Assume 500M daily active users. Average photo size after compression: 200KB. Average video size after transcoding: 2MB for a 30-second clip. Ratio of photos to videos: 80/20. Uploads per user per day: 0.5 (most users aren't power posters). That's 250M photos (50TB) and 62.5M videos (125TB) daily. Total: 175TB/day raw storage. Before replication. Before cold archival.

Bandwidth? On read-heavy workloads like feed generation, you serve 10x what you ingest. Add in thumbnail generation (multiple resolutions per upload), and your egress from object store to CDN to user is roughly 2PB/day. You do not handle that with a single region or a naive storage strategy. You design for it from day one.

// io.thecodeforge — interview tutorial

def estimate_daily_ingest(daily_active_users: int, uploads_per_user: float = 0.5,
                          photo_fraction: float = 0.8, avg_photo_kb: int = 200,
                          video_fraction: float = 0.2, avg_video_mb: int = 2) -> dict:
    total_uploads = int(daily_active_users * uploads_per_user)
    photos = int(total_uploads * photo_fraction)
    videos = total_uploads - photos

    photo_storage_gb = (photos * avg_photo_kb) / (1024 * 1024)
    video_storage_gb = (videos * avg_video_mb)

    return {
        "total_daily_uploads": total_uploads,
        "total_storage_gb": round(photo_storage_gb + video_storage_gb, 2),
        "estimated_egress_gb": round((photo_storage_gb + video_storage_gb) * 10, 2)
    }

capacity = estimate_daily_ingest(500_000_000)
print(f"Daily storage: {capacity['total_storage_gb']} GB")
print(f"Daily egress: {capacity['estimated_egress_gb']} GB")

System Requirements: The Black Box Your Interviewer Actually Opens

Competitor pages list Instagram features like a grocery list: "Post photos, follow users, like posts." Bored. Useless. The interviewer doesn't care if you can list features — they care if you can identify which ones create architectural tension. Here's the real distinction:

Core vs. Edge requirements. Posting photos is core. Creating a 4K video reel and expecting sub-second playback latency? That's a hard edge requirement that forces content-addressable storage and adaptive bitrate streaming. Follow/unfollow is easy until you have 10 million followers on a single celebrity account — then your fan-out model breaks.

Non-functional requirements are where the fight lives. Latency: feed generation must complete under 500ms for 99th percentile. Availability: 99.99% for read paths, but writes can tolerate brief inconsistency. Consistency: eventual consistency is fine for feed — users won't notice a 2-minute lag. But likes on a post? That expects real-time increment visibility. Different tradeoffs.

Capacity estimation isn't a separate section — it's how you validate every other choice. Store photos in S3? Great, what's your upload throughput per second? 100M daily uploads / 86400 seconds ≈ 1,157 uploads/second peak, likely 5x that during a viral event. Your upload API needs to handle 6,000 req/s with graceful degradation. That drives autoscaling policy, connection pooling, and database write throughput calculations.

// io.thecodeforge — interview tutorial

def estimate_write_throughput(daily_uploads: int, peak_multiplier: float = 5.0):
    seconds_per_day = 86400
    avg_throughput = daily_uploads / seconds_per_day
    # Assume peak hour is 5x average (Instagram's pattern)
    peak_throughput = avg_throughput * peak_multiplier
    return {
        "average_writes_per_second": round(avg_throughput, 2),
        "peak_writes_per_second": round(peak_throughput, 2),
        "peak_p95_percentage": f"{peak_multiplier * 100}% of average"
    }

io_requirements = estimate_write_throughput(100_000_000)
print(f"Avg writes/sec: {io_requirements['average_writes_per_second']}")
print(f"Peak writes/sec: {io_requirements['peak_writes_per_second']}")

Sharding Keys: The Decision That Makes or Breaks Your Database

Competitors vaguely mention 'database sharding' but never show you the knife fight: choosing the shard key. Instagram's core tables — User, Post, Follow, Like — each require different sharding strategies, and picking wrong means data hotspots that no cache can fix.

For the Post table, don't shard on user_id. Why? A celebrity posts once and gets 10M likes in an hour. That's 10M writes to a single shard. You've just created a burning hot partition. Instead, shard on post_id (or a hash of post_id). Spreads writes evenly because each post gets its own shard. Reads? Feed generation hits many post_ids anyway, so scatter-gather is unavoidable — but you parallelize that across shards.

For the Follow table? User_id shard works — a user's follow list is read-heavy and small (<10K follows, typically). Hot shard problem exists for celebrities (10M followers), but the read pattern is one user reading their follow graph, not 10M writes. Use a separate adjacency list in a cache layer like Redis for celebrity follow graph lookups. Don't torture your primary database.

For Likes, shard on (post_id, user_id) composite key. The goal is to evenly distribute like events across shards while supporting fast UNIQUE checks to prevent duplicate likes. Composite keys pre-split the write load. If you use auto-increment IDs, wrap them with a shard ID prefix — standard Ticket Server pattern.

// io.thecodeforge — interview tutorial

import hashlib

class ShardRouter:
    def __init__(self, num_shards: int = 64):
        self.num_shards = num_shards

    def _hash_to_shard(self, key: str) -> int:
        # Consistent hashing avoids reshuffling on shard count change
        return int(hashlib.md5(key.encode()).hexdigest(), 16) % self.num_shards

    def post_shard(self, post_id: str) -> int:
        return self._hash_to_shard(post_id)

    def like_shard(self, post_id: str, user_id: str) -> int:
        # Composite key for even distribution
        return self._hash_to_shard(f"{post_id}:{user_id}")

router = ShardRouter(num_shards=64)
print(f"Post 'abc123' -> shard {router.post_shard('abc123')}")
print(f"Like (post='abc123', user='user_99') -> shard {router.like_shard('abc123', 'user_99')}")

Interactions: The Hot Path You Will Mistake for a Side-Show

Interviews focus on feed reads but the hot path is writes from interactions. Like, comment, share, save — these hit the write-heavy chunk of your architecture every time a user breathes. You get 10x more interaction writes than uploads. And they need immediate consistency because no one wants to see their like disappear after a refresh.

Your read-replica strategy breaks here. Interaction writes must hit the primary shard containing the post owner's data. If you shard by user_id, the post's interaction traffic fans out across shards — every 'like' becomes a cross-shard transaction waiting to fail. Instead, shard interactions by post_id, colocating the post's metadata and interaction counters on the same node. This turns a multi-shard nightmare into a local increment.

The trap is treating interactions as low-priority. They generate the feed's engagement signals, trigger real-time notifications, and feed the ranking algorithm. If your design can't handle 500 concurrent likes on a single post, your Instagram clone dies on the first viral photo.

// io.thecodeforge — interview tutorial

class InteractionService:
    def like_post(self, user_id: str, post_id: str):
        # shard by post_id to keep metadata colocated
        post_shard = self.shard_for_post(post_id)
        
        # atomic increment on the primary, no cross-shard
        with post_shard.transaction():
            post_shard.increment('like_count', post_id)
            post_shard.insert('like', {
                'user_id': user_id,
                'post_id': post_id,
                'timestamp': self.now()
            })
        
        # fire async to feed ranking
        self.queue.send('interaction_rank', {
            'post_id': post_id,
            'type': 'like'
        })

User_Follows: The Graph Problem Your NoSQL Can't Solve

Your SQL fanboy starts talking about a join table. Wrong. Instagram has 500M+ daily active users. A user follows 500 accounts. That's 250B edges. joins at that scale are a death sentence. You need a directed graph stored as a lookup table in a document store or a specialized graph database like Amazon Neptune or Neo4j.

But here's the real problem: when a post goes viral, your feed generation needs to check 'does user A follow user B?' This check happens per post, per feed request, for every user scrolling. A simple key-value check — user_id -> set of followee_ids — handles this at sub-millisecond latency. Store that in Redis as a sorted set. Use the follow timestamp as the score so the feed generator can paginate by recency without a costly join.

The follow/unfollow operation is another trap. Stale followers in the cache cause phantom content in the feed. Use a write-through cache pattern: on unfollow, delete the cached following set for that user. Then lazy-load it on the next feed request. This keeps consistency without cascading invalidations across nodes.

// io.thecodeforge — interview tutorial

class FollowGraph:
    def __init__(self, redis, db):
        self.cache = redis
        self.store = db
    
    def get_following(self, user_id: str) -> set:
        # check cache first
        cached = self.cache.smembers(f"following:{user_id}")
        if cached:
            return cached
        
        # lazy load from DB if cache miss
        following = self.store.query(
            "SELECT followee_id FROM follows WHERE follower_id = ?", user_id
        )
        self.cache.sadd(f"following:{user_id}", *following)
        self.cache.expire(f"following:{user_id}", 3600)
        return following
    
    def unfollow(self, follower: str, followee: str):
        self.store.execute(
            "DELETE FROM follows WHERE follower_id = ? AND followee_id = ?",
            follower, followee
        )
        # write-through cache invalidation
        self.cache.delete(f"following:{follower}")

High-Level Design (HLD): The Diagram That Defines Your Interview

Your HLD is not a network topology. It is a story about data flow under constraints. Start with the read vs. write asymmetry: 100M uploads/day vs. 500M feed refreshes/day. Every component choice follows from that ratio. Use a single API gateway for authentication, rate limiting, and request routing. Separate upload and feed read paths at the gateway level—uploads hit a dedicated ingestion service, feeds hit a read-optimized service. The fan-out service writes to each follower's feed cache (push) while a lazy loader fills gaps for inactive users (pull). Your HLD must show the reverse index for comments and likes as separate logical stores, even if physically shared. The interviewer wants to see you encode trade-offs—like why you pick Kafka over RabbitMQ for the async fan-out (throughput, replayability) or why you split the feed read service from the user service (independent scaling, failure isolation). A strong HLD for Instagram is a decision tree, not a checklist.

// io.thecodeforge — interview tutorial

# HLD decision rules translated to code logic

class FeedHLD:
    def __init__(self):
        self.read_write_ratio = 5_000_000_000 / 100_000_000  # 50:1
        self.use_kafka = True  # async fan-out needs replay

    def choose_gateway_split(self):
        return {"upload": "ingestion_service", "feed": "read_service"}

    def scaling_rules(self):
        # only scale read service horizontally
        return "feed_read instances >> upload instances"

Low-Level Design (LLD): The Feed Cache That Must Survive Queries per Second

Feed generation at scale fails in data structures, not algorithms. Your LLD must specify the exact Redis data types for the feed cache: a Redis sorted set per user where the key is post_id and score is creation timestamp. Each post stores a serialized object with media_id, user_id, and like_count (not the full image URL—that's in CDN). The fan-out worker writes to these sets in batch: for each new post, it inserts into the top 10,000 followers' sorted sets (push), then signals a lazy loader thread for the rest. The lazy loader uses a Bloom filter to check if a post was already pushed to a user's cache before falling back to the follower's SQL index. Eviction policy is TTL-based (72 hours for active users, 24 for dormant). Cache misses warm from a denormalized Postgres table with index on (user_id, created_at DESC). Your LLD must also handle likes and comments as counter shards: each post's interaction count lives in a Redis hash updated asynchronously via Kafka streams, avoiding write amplification on the feed read path.

// io.thecodeforge — interview tutorial

import redis

r = redis.Redis(decode_responses=True)

def push_post_to_feed(follower_id, post_id, timestamp):
    # sorted set: key = feed:{user_id}, score = timestamp
    r.zadd(f"feed:{follower_id}", {post_id: timestamp})

def get_feed(user_id, start, end):
    # reverse range, newest first
    return r.zrevrange(f"feed:{user_id}", start, end, withscores=True)

# Bloom filter check before lazy load
# store follower count per post to decide push vs pull