Design Amazon — S3 Blast Radius and Checkout Races

Amazon processes over 66,000 orders per minute at peak, serves hundreds of millions of customers across 20+ countries, and runs one of the most complex distributed systems ever built — all while most transactions complete in under a second. Understanding how to design a system at this scale isn't just an interview exercise; it's a masterclass in the real trade-offs that define modern software engineering: consistency vs. availability, latency vs. accuracy, operational simplicity vs. raw performance.

The core problem Amazon solves is multi-dimensional. It's not just a database with a shopping cart on top. It's a real-time inventory system, a personalization engine, a payments processor, a logistics orchestrator, a search engine, and a seller marketplace — all running simultaneously, all needing to agree on the state of the world, and all needing to survive individual component failures without the customer ever noticing. The challenge isn't writing any one of these systems; it's making them work together under crushing load.

By the end of this article, you'll be able to walk into a system design interview and articulate a coherent, production-realistic Amazon architecture. You'll understand why the product catalog is separated from inventory, why the cart lives in a different data store than order history, how search is decoupled from the relational database, and what actually happens between you clicking 'Buy Now' and your order appearing on screen. You'll know the real trade-offs, not just the happy path.

Core Architecture Principles

Amazon's architecture is built on a few non-negotiable principles. First, data ownership is absolute: each microservice owns its data exclusively — no shared tables between services. Second, communication is asynchronous where possible: use events (Kafka) for order creation, inventory updates, and shipping triggers. Synchronous calls are reserved for operations that need immediate confirmation, like payment gateway interaction. Third, cache everything that can be stale. The product catalog, search results, recommendations — all served from cached layers that accept minutes of staleness. Fourth, fail gracefully: if a downstream service is down, the system degrades, it doesn't crash. The homepage might show fewer recommendations, but the site stays up.

These principles are not theoretical — they were earned through real production failures. The 2017 S3 outage showed that shared infrastructure can bring down the entire site. The 2020 DynamoDB throttling event during Prime Day taught them to provision for 3x peak traffic. Every principle has a scar.

package io.thecodeforge.order;

import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

@Component
public class EventPublisher {
    private final KafkaTemplate<String, OrderEvent> kafka;

    public EventPublisher(KafkaTemplate<String, OrderEvent> kafka) {
        this.kafka = kafka;
    }

    public void publishOrderCreated(Order order) {
        OrderEvent event = new OrderEvent(
            order.getId(),
            order.getCustomerId(),
            order.getTotal(),
            "CREATED"
        );
        kafka.send("order_events", order.getId(), event)
            .whenComplete((result, ex) -> {
                if (ex != null) {
                    // Fallback: log to dead letter queue or retry
                    log.error("Failed to publish order event for {}", order.getId(), ex);
                }
            });
    }
}

Requirements & Estimation

Before drawing boxes, we need numbers. Amazon serves ~200M active customers, processes ~66,000 orders/min at peak. Every second, that's ~1,100 orders. Each order generates writes to cart, inventory, payment, order, and logistics services. Read-to-write ratio for the product catalog is roughly 100:1, while for cart it's 1:1 (every add is followed by a read during checkout). Storage: product catalog ~100M items, each with 10-50 KB metadata — that's ~5 TB in the DB. Images stored in object storage (S3), total petabyte-scale. Network bandwidth: each page load transfers ~2 MB (HTML, JS, images). At 200M DAU, average 10 pages per session = 2B page loads/day = ~4 PB/day outbound — that's why you need CDN and aggressive caching.

These numbers drive every architecture decision. You don't design for 66K orders/min without knowing your bottleneck: database writes per second, queue throughput, payment latency SLA. A common mistake: designing for average load, not peak. Prime Day traffic spikes 5-10x above average. So you need to provision for at least 2x your estimated peak, and then use auto-scaling to handle surges.

Daily active users: 200M
Orders per second (peak): ~1,100
Reads/sec on catalog DB: ~2,000,000 (200M * 10 pages / 86400 seconds)
Writes/sec to order DB: ~1,100 (plus cart, inventory, payment)
Storage: catalog 5 TB, images ~50 TB
CDN bandwidth: ~40 Gbps at peak (2MB * 2B page loads / 86400)

High-Level Architecture — Service Decomposition

Amazon's architecture is a collection of hundreds of microservices. The core ones for an e-commerce platform:

• Product Catalog Service: read-heavy, exposes product details, categories, images. Uses a read replica with a CDN cache for images.
• Inventory Service: tracks stock per warehouse. Must be strongly consistent to avoid overselling. Usually a separate database (Aurora with row-level locking).
• Cart Service: low-latency, high-write. Uses DynamoDB with eventual consistency for add/remove operations; cart read during checkout uses strong consistency.
• Order Service: receives checkout request, orchestrates the saga: reserve inventory, process payment, create order, trigger shipping. Uses a queue for decoupling.
• Payment Service: idempotent, integrates with external gateways. Stores transaction logs in a relational DB.
• Search Service: Elasticsearch cluster indexed from catalog and inventory changes via CDC.
• Recommendation Service: ML pipeline producing real-time recommendations served via a separate read-optimised cache.
• Shipping Service: async, watches order completion events and sends to logistics.

Each service has its own database, communicates via HTTP/REST or async events (Kafka). API Gateway routes requests, handles authentication, rate limiting.

Client -> CloudFront (CDN) -> API Gateway -> Services:
  - Product Catalog (Aurora, Redis cache)
  - Inventory (Aurora, strong consistency)
  - Cart (DynamoDB)
  - Order (Aurora + SQS/Kafka)
  - Payment (Aurora, external gateway via idempotent HTTP)
  - Search (Elasticsearch)
  - Recommendations (Redis / ML predictions)
  - Shipping (DynamoDB + SQS -> logistics)
Each service owns its DB. Async events flow through Kafka topics (order_created, payment_completed, etc.)

Data Consistency & Trade-offs Across Services

Amazon must maintain consistency where it matters (inventory, payments) and accepts eventual consistency where it doesn't (product catalog updates, recommendations). The key trade-offs:

• Product Catalog: writes are rare (admin updates), reads are massive. Use a leaderless read-replica architecture with cache-aside pattern. A catalog update can take minutes to propagate to all edge caches — that's fine.
• Inventory: overselling is unacceptable. When a customer adds an item to cart, we reserve inventory for 15 minutes. If not checked out, the reservation expires. This is optimistic — but during high contention, we risk deadlocks. Use row-level locking in Aurora for the inventory row. This limits throughput to ~1000 inventory reservations per second per row. Solution: shard inventory by product ID (each product gets its own partition).
• Cart & Order: The cart service uses eventual consistency for add/remove, but during checkout, the order service reads the cart with strong consistency and then runs a saga: reserve inventory (idempotent), charge payment (idempotent), decrement inventory, create order. If any step fails, compensate: release inventory, void payment.
• Search: Elasticsearch is eventually consistent with the inventory DB. If you add an item, it might take seconds to appear in search results. Acceptable for most queries, but for sellers pushing inventory updates, we provide a synchronous fallback: if a seller uses 'update inventory API', we directly update a cache that search reads with low latency.

Use gossip protocols and CRDTs where possible for coordination-free eventual consistency.

package io.thecodeforge.order;

import java.util.UUID;

public class PaymentSaga {
    private static final String IDEMPOTENCY_KEY_HEADER = "X-Idempotency-Key";

    // This method is called by the order saga orchestrator
    public SagaResult processPayment(String orderId, double amount) {
        String idempotencyKey = UUID.randomUUID().toString(); // stored in DB per order
        PaymentRequest request = new PaymentRequest(orderId, amount, idempotencyKey);
        try {
            PaymentResponse response = paymentGateway.charge(request);
            if (response.isSuccess()) {
                return SagaResult.success(orderId, response.getTransactionId());
            } else {
                // Compensate: release inventory
                inventoryService.releaseReservation(orderId);
                return SagaResult.failure(orderId, "Payment declined");
            }
        } catch (TimeoutException e) {
            // Idempotency ensures retry doesn't double-charge
            // log and schedule retry with same idempotency key
            return SagaResult.retryLater(orderId, idempotencyKey);
        }
    }
}

Search & Recommendations — The Read-Optimised Path

Search and recommendations are the two features with the highest read load on Amazon. Both are served entirely from caches and search indices, never touching the main OLTP databases.

Search: Users type a query, API Gateway routes to Search Service, which queries Elasticsearch (ES). ES returns product IDs, then the service fetches product details from a local Redis cache (or falls back to catalog DB). The search index is updated asynchronously via Kafka connect from the inventory and catalog databases. Latency target: under 100ms P99.

Recommendations: For each page load, the frontend sends user context (user ID, page category, recent searches). The Recommendation Service runs an ML model (e.g., collaborative filtering with matrix factorisation) tuned every 6 hours. Model outputs are pre-computed for each user and stored in Redis with a TTL of 12 hours. The service returns a list of product IDs, and the frontend fetches details from the same cache layer as search. Latency target: under 50ms P99.

To scale search, we use a tiered approach: popular queries are cached in a local CDN node (Varnish) with 5-minute TTL. Hot product details are in Redis with sharding across nodes. Cold products go to Elasticsearch with a larger shard count.

// TheCodeForge — Search query flow
// 1. Client sends query to API Gateway
// 2. API Gateway checks CDN cache for this query
// 3. Cache miss -> route to Search Service
// 4. Search Service queries Elasticsearch with:
{
  "query": {
    "bool": {
      "must": [
        { "match": { "product_name": "wireless headphones" } },
        { "term": { "in_stock": true } }
      ]
    }
  },
  "size": 20,
  "_source": ["product_id"]
}
// 5. Get list of product IDs
// 6. Fetch full product details from Redis (batch get)
// 7. Return combined response to client

Caching & CDN Strategy

Amazon's read volume is staggering — millions of requests per second for catalog pages, images, search results. Without a multi-tier caching strategy, the origin databases would collapse. The caching layers, from edge to database:

1. CDN (CloudFront): Caches static assets (product images, CSS, JS) at edge locations. TTL of 24 hours for assets, invalidated on new uploads. For dynamic content (search results, recommendations), CDN caches only popular queries with short TTL (5 minutes).
2. API Gateway Cache: Regional cache for identical API responses. Works well for product details that don't change often.
3. Service-level Cache (Redis): Each service has its own Redis cluster. Catalog service caches product details by ID (LRU eviction). Cart service uses Redis for session data. Recommendation service caches precomputed user recommendations.
4. Database Read Replicas: Aurora read replicas handle cache misses. In extreme cases, they can be promoted to handle more read load.

The design principle: the top 5% of hottest products receive 80% of traffic (Pareto). Cache those aggressively. Long-tail products are served from Elasticsearch or read replicas with lower priority.

package io.thecodeforge.cache;

public class CatalogService {
    private final RedisCache cache;
    private final ProductRepository db;

    public Product getProduct(String productId) {
        Product cached = cache.get(productId);
        if (cached != null) {
            return cached;
        }
        Product fromDb = db.findById(productId);
        if (fromDb != null) {
            cache.set(productId, fromDb, Duration.ofMinutes(10));
        }
        return fromDb;
    }

    public void updateProduct(Product product) {
        db.save(product);
        cache.invalidate(product.getId());
        // Also send event to search service to reindex
        eventPublisher.publishProductUpdated(product);
    }
}

Checkout Flow — From Cart to Confirmation

When the user clicks 'Place Order', this is the most critical path. Here's the real sequence:

1. Cart Service retrieves the user's cart with strong consistency (gets latest items and their IDs).
2. Order Service receives the checkout request and starts a saga:
3. - Reserve Inventory: for each item, call Inventory Service to reserve quantity. If any item is insufficient, fail the entire order (release other reservations).
4. - Process Payment: call Payment Service with the total amount and an idempotency key. The payment service interacts with the external gateway. If timeout, retry (idempotency prevents double charge).
5. - Create Order: insert order record into Order DB.
6. - Decrement Inventory: final decrement of reserved quantities.
7. - Send to Shipping: publish order_created event to Kafka, which the Shipping Service picks up.
8. If any step fails after payment, a compensation transaction is run: refund payment, release remaining inventory. This compensation is also idempotent.
9. The frontend polls the Order Service for the order status (every 2 seconds until confirmed) and then redirects to the order confirmation page.

All services use asynchronous communication where possible to reduce end-to-end latency. The entire saga typically completes in under 500ms for 95% of orders.

Client -> API Gateway: POST /checkout
  -> Order Service: start saga
    -> Inventory Service: reserve(item1, qty1), reserve(item2, qty2)
    <- OK all reserved
    -> Payment Service: charge(amount, idempotency_key)
    <- OK transaction_id
    -> Order Service: insert order (status: CONFIRMED)
    -> Inventory Service: decrement(item1, qty1), decrement(item2, qty2)
    -> Kafka: emit OrderCreated event
  <- 200 OK

Compensation path (if payment fails after inventory reserve):
  -> Inventory Service: releaseReservation(orderId)
  -> Order Service: update order status to FAILED