Notification System Design — Silent SMS Soft-Ban Traps

Every time you get a payment confirmation from your bank, a 'your package shipped' email from Amazon, or a red badge on your Instagram icon, a notification system fired behind the scenes. These systems are invisible when they work and catastrophic when they don't — a failed OTP SMS locks a user out of their account, a duplicate push notification at 3 AM turns a loyal customer into a one-star reviewer. Notification systems are deceptively simple on the surface and brutally hard in production.

The core problem a notification system solves is decoupling event producers from delivery channels. Your payment service shouldn't need to know whether a user prefers SMS, email, or push — and it definitely shouldn't block waiting for Twilio to respond. The notification system absorbs that complexity: it stores user preferences, throttles sends, fans out to multiple channels, retries failures, and records delivery receipts — all asynchronously and at scale.

By the end of this article you'll be able to design a notification system that handles 10 million daily active users across email, SMS, and push — including the fan-out architecture, idempotency strategy, rate limiting design, and the five production edge cases that trip up even experienced engineers. You'll also have the vocabulary and depth to walk through this confidently in a senior system design interview.

Functional & Non-Functional Requirements

Before designing any system, define what it must do and how well it must do it. For a notification system, functional requirements include the ability to send notifications via email, SMS, and push; support multiple languages and templates; respect user preferences (opt-in/out); and track delivery status. Non-functional requirements are where senior engineers focus: latency (under 1 second for critical OTPs, up to 5 minutes for promotional emails), throughput (10 million notifications/day), availability (99.99% uptime), and durability (never lose a notification once accepted). The biggest mistake? Treating all notifications with the same priority. An OTP failure is a revenue and security incident; a marketing email failure is a minor miss. Your architecture must distinguish between them.

High-Level Architecture: Event Ingestion to Delivery

A notification system can be broken into four layers: ingestion, processing, delivery, and tracking. Ingestion receives an event from any service (e.g., PaymentService sends a 'payment_confirmed' event). The event enters a message broker like Kafka. The processing layer reads events, enriches them with user preferences and templates, decides which channels to use, and applies rate limiting. Then it creates individual notification tasks per channel and pushes them to channel-specific queues. The delivery layer consumes those tasks, calls third-party providers (Twilio for SMS, SendGrid for email, Firebase for push), and stores the result. Tracking layer captures delivery receipts, opens, clicks, and bounces. This architecture decouples producers from delivery, allows independent scaling of each layer, and provides a single place to add monitoring, retries, and compliance.

Fan-Out Strategies: Direct, Batch, and Tiered

Fan-out is how one event becomes many deliveries. The simplest approach is direct fan-out: for each event, the processing layer queries the user preference store and creates one notification task per subscribed channel. This works for low volumes but becomes expensive at scale — you're doing N database lookups per event. Batch fan-out groups users by channel and preference, reducing lookups. For example, a 'promotion' event might define a target segment (e.g., all users in tier 'premium'), and the system generates email tasks in bulk without individual lookups. Tiered fan-out combines both: critical events use direct (fast, per-user), bulk events use batch. The trade-off is latency vs. throughput. Direct fan-out is O(1) per event per user; batch fan-out is O(1) per segment. Choose based on the notification type's SLA.

Reliability, Retries, and Dead-Letter Queues

Third-party providers fail — they return 5xx, throttle you, or go down. Your notification system must handle these failures gracefully. The standard pattern: exponential backoff with jitter (e.g., 1s, 4s, 16s, 64s max), configurable max retries per channel and per priority. After exhausting retries, move the task to a dead-letter queue (DLQ). The DLQ stores failed notifications for manual inspection or later reprocessing. Crucially, retries must be idempotent: the same retry should not send two SMS messages. Use a unique idempotency key (e.g., eventId + channel) stored in Redis with a TTL longer than the retry window. For example, if max retries took 5 minutes, set Redis TTL to 30 minutes. Also, the DLQ must trigger an alert — a silent DLQ means you're losing notifications.

Production Gotchas: Rate Limiting, Deduplication, and Channel Failures

Three non-obvious issues that bring down notification systems:

1. Rate limiting at the provider level: SMS providers like Twilio impose per-second and per-day limits. Exceeding them triggers soft bans (HTTP 200 but no delivery). Solution: implement a token bucket rate limiter per provider and per channel. Monitor the token consumption against the limit.
2. Deduplication across retries and chains: If a user receives a notification that triggers another notification (e.g., 'payment received' triggers 'transaction alert'), you need cross-event deduplication. Use a configurable suppression window (e.g., don't send same type within 5 minutes). Store suppression keys in Redis with TTL.
3. Channel failures: Email domain may have strict SPF/DKIM, causing bounces. Push notification certificates expire. SMS aggregator may be down in a region. Solution: have a fallback channel strategy (e.g., if email fails, send SMS). Define fallback rules per notification type. Monitor channel health with synthetic probes.

Frequently Asked Questions