Design Spotify: A Production System Architecture That Handles 500M Users

You're building a music streaming service. Day one, it works fine. Day 100, your database is on fire because you stored playlists in a relational database with joins. I've seen this exact pattern kill a startup's launch. The problem isn't the music — it's the metadata. Playlists, user history, social features — these are graph-like, write-heavy, and need to be available globally. Spotify handles 500M users with 100M+ tracks. They don't use a single database. They use a carefully chosen set of tools, each optimized for a specific job. After this article, you'll be able to design a system that scales to millions of users without falling over.

Why Microservices? The Monolith That Couldn't Stream

Spotify started as a monolith. By 2012, it was a nightmare. Deploying a change to the recommendation engine required redeploying the entire backend. A bug in playlist sharing could take down search. They split into microservices — each owning a bounded context: playlist service, social graph service, audio delivery service, search service, etc. This let them scale independently. The playlist service, for example, is write-heavy and uses Cassandra. The search service is read-heavy and uses Elasticsearch. They communicate via gRPC for low latency and Kafka for async events. The trade-off? Distributed debugging is harder. You need tracing (Jaeger) and structured logging. But for a system with 500M users, the isolation is worth it.

// io.thecodeforge — System Design tutorial

// High-level service interaction for adding a track to a playlist

// Client -> API Gateway -> Playlist Service
// Playlist Service writes to Cassandra, publishes event to Kafka
// Kafka event consumed by Search Service (to index), Recommendation Service (to update model)

// This is a sequence diagram in text:
// 1. Client POST /playlists/{id}/tracks
// 2. API Gateway authenticates, routes to Playlist Service
// 3. Playlist Service validates track exists (calls Metadata Service via gRPC)
// 4. Playlist Service writes to Cassandra: INSERT INTO playlist_tracks (playlist_id, track_id, added_at, position) VALUES (?, ?, ?, ?)
// 5. Playlist Service publishes event to Kafka topic 'playlist_updates'
// 6. Search Service consumes event, updates Elasticsearch index
// 7. Recommendation Service consumes event, updates user's listening history in Cassandra

// Key: gRPC for synchronous calls (low latency), Kafka for async (decoupling)

Cassandra for Playlists: Why Not SQL?

Playlists are write-heavy. Users add/remove tracks, reorder, share. A relational database with normalized tables (playlists, playlist_tracks, users) would require joins on every read. With millions of playlists and billions of tracks, joins become a bottleneck. Cassandra is a wide-column store that excels at write throughput and can handle large datasets with no single point of failure. The data model: playlist_tracks table with partition key playlist_id and clustering columns (position, track_id). This allows fast range queries for a playlist's tracks. The downside? No joins, no transactions across partitions. You have to denormalize. For example, store playlist metadata (name, owner) in a separate table with the same partition key. Spotify also uses Cassandra for user library (saved tracks, albums) and listening history. The trade-off is eventual consistency — but for playlists, that's acceptable. If a user adds a track and it doesn't appear immediately on another device, it's fine.

// io.thecodeforge — System Design tutorial

// Cassandra schema for playlists

CREATE TABLE IF NOT EXISTS playlist_tracks (
    playlist_id uuid,
    position int,
    track_id text,
    added_at timestamp,
    added_by text,
    PRIMARY KEY (playlist_id, position)
) WITH CLUSTERING ORDER BY (position ASC);

// Query: get all tracks in a playlist ordered by position
SELECT * FROM playlist_tracks WHERE playlist_id = ? ORDER BY position ASC;

// Insert a track at the end
INSERT INTO playlist_tracks (playlist_id, position, track_id, added_at, added_by)
VALUES (?, ?, ?, toTimestamp(now()), ?);

// Note: position is managed by the application. To insert at a specific position, shift subsequent positions.
// This is a batch operation within the same partition (Cassandra supports lightweight transactions for idempotency).

// For playlist metadata:
CREATE TABLE IF NOT EXISTS playlists (
    playlist_id uuid PRIMARY KEY,
    name text,
    owner_id text,
    description text,
    created_at timestamp,
    updated_at timestamp
);

Audio Streaming: The CDN and Adaptive Bitrate

Streaming audio is the core of Spotify. They don't serve audio from their own servers — that would be insane. They use a CDN (Google Cloud CDN, Akamai, etc.) to cache audio files close to users. Audio files are encoded in multiple bitrates (96kbps, 160kbps, 320kbps Ogg Vorbis). The client selects the appropriate bitrate based on network conditions (adaptive bitrate streaming). The challenge: how does the client know which CDN edge to hit? Spotify uses a 'delivery service' that returns a signed URL to the closest CDN edge. The URL includes a token for authentication. The client then fetches the audio file directly from the CDN. This offloads traffic from Spotify's infrastructure. The CDN handles the heavy lifting of global distribution. The trade-off: cost. CDN egress is expensive. Spotify optimizes by caching popular tracks aggressively and using peer-to-peer for some scenarios (though that's less common now).

// io.thecodeforge — System Design tutorial

// Client requests audio stream
// 1. Client sends GET /stream/{track_id} to API Gateway
// 2. API Gateway authenticates, calls Delivery Service
// 3. Delivery Service checks user's subscription (free vs premium) to determine bitrate
// 4. Delivery Service generates a signed URL: https://cdn.spotify.com/audio/{track_id}.ogg?token={jwt}&expires={timestamp}
// 5. Client receives URL, fetches from CDN
// 6. CDN checks if file is cached. If not, fetches from origin storage (Google Cloud Storage)
// 7. Client plays audio, requests next chunk if needed

// Key: signed URL prevents unauthorized access. Expiry time is short (e.g., 1 hour).
// For live streaming, use HLS (HTTP Live Streaming) with .m3u8 playlist.

Recommendations: The Offline Pipeline That Never Sleeps

Spotify's recommendation engine is the secret sauce. It's not real-time — it's a batch processing pipeline that runs periodically (daily or hourly). It uses collaborative filtering, natural language processing (on song lyrics, blog posts), and audio analysis (tempo, key, loudness). The pipeline is built on Apache Beam (or Scio, a Scala wrapper for Beam) running on Google Cloud Dataflow. It processes user listening history (stored in Cassandra) and track features (stored in Bigtable) to generate recommendations. The output is stored in a 'recommendation table' in Cassandra, keyed by user_id. When a user opens the app, the client fetches recommendations from this table. The trade-off: recommendations are not real-time. If a user listens to a new genre, it won't affect recommendations until the next pipeline run. But that's acceptable — users don't expect instant personalization.

// io.thecodeforge — System Design tutorial

// Pseudo-code for recommendation pipeline (simplified)

// 1. Read user listening history from Cassandra (last 30 days)
// 2. For each user, compute vector of liked tracks (weighted by play count, skip rate)
// 3. Use collaborative filtering to find similar users
// 4. For each user, find tracks liked by similar users that the user hasn't heard
// 5. Rank tracks by predicted affinity score
// 6. Write top 50 recommendations to Cassandra: INSERT INTO recommendations (user_id, track_id, score) VALUES (?, ?, ?)

// This runs as a Dataflow pipeline with sliding windows.
// For real-time personalization (e.g., Discover Weekly), use a separate pipeline that updates daily.

Search: Elasticsearch at Scale

Search is critical. Users search for tracks, albums, artists, playlists. Spotify uses Elasticsearch with a custom scoring function that combines text relevance (BM25) with popularity signals. The index is sharded across multiple nodes. Updates come from Kafka events (when a new track is added, metadata changes). The challenge: keeping the index fresh. Spotify uses near-real-time indexing with a refresh interval of 1 second. For large updates (e.g., new album), they use bulk indexing. The trade-off: Elasticsearch is memory-hungry. Each shard consumes heap. They optimize by using doc-values for aggregations and avoiding heavy analyzers on large fields.

// io.thecodeforge — System Design tutorial

// Elasticsearch mapping for tracks
PUT /tracks
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1,
    "refresh_interval": "1s"
  },
  "mappings": {
    "properties": {
      "track_name": { "type": "text", "analyzer": "standard" },
      "artist_name": { "type": "text", "analyzer": "standard" },
      "album_name": { "type": "text", "analyzer": "standard" },
      "popularity": { "type": "integer" },
      "duration_ms": { "type": "integer" },
      "genre": { "type": "keyword" },
      "release_date": { "type": "date" }
    }
  }
}

// Query: search for tracks by name or artist
GET /tracks/_search
{
  "query": {
    "multi_match": {
      "query": "bohemian rhapsody",
      "fields": ["track_name^3", "artist_name^2", "album_name"],
      "type": "best_fields"
    }
  },
  "sort": [
    { "popularity": "desc" }
  ]
}

Social Features: Graph Database for Friends and Follows

Spotify has social features: follow friends, see their playlists, collaborative playlists. This is a graph problem. They use a graph database (Neo4j) for the social graph. Why not relational? Because queries like 'find all friends of friends who listened to this track' are expensive in SQL with recursive joins. Neo4j handles these with ease. The social graph is write-heavy (follow/unfollow) but read-heavy for recommendations. They replicate the graph to Cassandra for read-heavy workloads (denormalized). The trade-off: maintaining consistency between Neo4j and Cassandra. They use an event-driven approach: when a follow happens, write to Neo4j, then publish event to Kafka, which updates Cassandra. This is eventually consistent, but acceptable.

// io.thecodeforge — System Design tutorial

// Neo4j Cypher query to find friends who listened to a track
MATCH (u:User {id: $user_id})-[:FOLLOWS]->(friend:User)-[:LISTENED]->(track:Track {id: $track_id})
RETURN friend.id

// This is efficient because Neo4j traverses relationships in constant time per hop.
// In SQL, this would require multiple joins or recursive CTEs.

// For write-heavy operations, use batch inserts:
UNWIND $relationships AS rel
MATCH (u:User {id: rel.follower}), (v:User {id: rel.followee})
MERGE (u)-[:FOLLOWS]->(v)

Event-Driven Architecture with Kafka

Kafka is the backbone of Spotify's async communication. Every significant action (play track, add to playlist, follow user) publishes an event to Kafka. Multiple consumers process these events: search indexing, recommendation pipeline, analytics, billing (for premium). This decouples services and allows independent scaling. The key design: use Avro for schema evolution. Spotify has a schema registry to ensure compatibility. They also use Kafka Streams for real-time processing (e.g., updating user session state). The trade-off: Kafka adds latency (milliseconds) and operational complexity. But for a system with hundreds of microservices, it's essential.

// io.thecodeforge — System Design tutorial

// Avro schema for track play event
{
  "type": "record",
  "name": "TrackPlayEvent",
  "namespace": "com.spotify.event",
  "fields": [
    {"name": "user_id", "type": "string"},
    {"name": "track_id", "type": "string"},
    {"name": "timestamp", "type": "long"},
    {"name": "context", "type": ["null", "string"], "default": null}, // e.g., "playlist:123"
    {"name": "source", "type": ["null", "string"], "default": null} // e.g., "search", "recommendation"
  ]
}

// Producer (in Python using confluent-kafka)
producer.produce('track_plays', value=avro_serializer(event, schema))

// Consumer (in Java)
KafkaConsumer<String, TrackPlayEvent> consumer = new KafkaConsumer<>(props);
consumer.subscribe(Arrays.asList("track_plays"));
while (true) {
    ConsumerRecords<String, TrackPlayEvent> records = consumer.poll(100);
    for (ConsumerRecord<String, TrackPlayEvent> record : records) {
        processPlayEvent(record.value());
    }
}

Caching: Redis for Hot Data

Spotify uses Redis extensively for caching: user sessions, recently played tracks, top charts, and playlist metadata for popular playlists. The key is to cache only hot data. For example, the top 1% of playlists account for 80% of reads. They cache these in Redis with a TTL of 5 minutes. For less popular playlists, they fall through to Cassandra. They also use Redis for rate limiting and distributed locks. The trade-off: cache invalidation is hard. They use a write-through cache: when a playlist is updated, they invalidate the Redis key and let the next read repopulate it from Cassandra. This ensures consistency at the cost of a cache miss.

// io.thecodeforge — System Design tutorial

// Pseudo-code for caching playlist tracks

// On read:
function getPlaylistTracks(playlistId):
    cached = redis.get("playlist:" + playlistId)
    if cached:
        return deserialize(cached)
    tracks = cassandra.execute("SELECT * FROM playlist_tracks WHERE playlist_id = ?", playlistId)
    redis.setex("playlist:" + playlistId, 300, serialize(tracks)) // TTL 5 min
    return tracks

// On write (add track):
function addTrackToPlaylist(playlistId, trackId):
    cassandra.execute("INSERT INTO playlist_tracks ...", playlistId, trackId)
    redis.del("playlist:" + playlistId) // invalidate cache
    // Also publish event to Kafka for search/recommendations

Monitoring and Observability

With hundreds of services, you need centralized logging, metrics, and tracing. Spotify uses Google Cloud Monitoring for metrics, Jaeger for distributed tracing, and ELK stack for logs. Every service exposes Prometheus metrics (request rate, latency, error rate). They have dashboards for each service and SLOs (e.g., playlist read latency p99 < 100ms). Tracing is critical for debugging latency issues across services. They use a correlation ID propagated via gRPC metadata. The trade-off: instrumentation adds overhead. They sample traces (1% of requests) to keep costs down. But when debugging an incident, they can increase sampling rate dynamically.

// io.thecodeforge — System Design tutorial

// In Go, using OpenTelemetry
import (
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/trace"
)

// Create a tracer
tracer := otel.Tracer("playlist-service")

// In a request handler
func handleGetPlaylist(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "getPlaylist")
    defer span.End()

    // Call downstream service with context
    metadataResp, err := metadataService.GetTrack(ctx, trackID)
    if err != nil {
        span.RecordError(err)
        http.Error(w, err.Error(), 500)
        return
    }
    // ...
}

// Propagate context via gRPC metadata
// The gRPC interceptor automatically extracts/injects trace context.