Design a Web Crawler: Build a Production Crawler That Won't Get You Banned

Most web crawler tutorials are toys. They show you how to fetch a page with requests and call it a day. In production, that naive approach gets your IP banned, crashes your database, and costs you thousands in bandwidth. I've seen a startup's entire crawling pipeline grind to a halt because they forgot to deduplicate URLs — they downloaded the same page 50,000 times. This article is the real deal: how to design a web crawler that respects robots.txt, handles rate limiting, scales to millions of pages, and doesn't get you sued. By the end, you'll be able to architect a distributed crawler that runs for weeks without manual intervention.

The Frontier: Your Crawler's Brain

The frontier is the queue of URLs to crawl. Without it, your crawler is a headless chicken. The naive approach is a FIFO queue, but that ignores politeness and priority. You need a priority queue that respects robots.txt crawl-delay and gives fresh content higher priority. I've seen teams use a simple Redis list and wonder why they get banned — because they hammer the same domain with 100 concurrent requests. The correct design: a set of per-domain queues, each with its own rate limiter. When a URL is discovered, hash the domain, push to that domain's queue. A scheduler picks the next domain whose rate limit allows a request. This ensures you never exceed the crawl-delay for any domain.

// io.thecodeforge — System Design tutorial

// Frontier using Redis per-domain queues
// Each domain has a sorted set keyed by crawl priority (timestamp of last crawl)
// Scheduler picks domain with oldest last-crawl time that has waited >= crawl-delay

// Pseudocode for scheduler:
function nextUrl() {
    while (true) {
        domain = redis.zpopmin('domains')  // get domain with oldest last-crawl
        if (domain == null) break
        lastCrawl = redis.get('lastcrawl:' + domain)
        delay = getCrawlDelay(domain)  // from robots.txt cache
        if (now - lastCrawl >= delay) {
            url = redis.lpop('queue:' + domain)
            if (url) {
                redis.set('lastcrawl:' + domain, now)
                return url
            }
        } else {
            // domain not ready, push back with updated score = lastCrawl + delay
            redis.zadd('domains', lastCrawl + delay, domain)
        }
    }
    sleep(100ms)
}

Politeness: Don't Be a Jerk

Politeness is the single most important aspect of a production crawler. Ignore it and you'll get IP-banned, blocked by Cloudflare, or sued. The rules: respect robots.txt, obey crawl-delay, and throttle to a reasonable rate per domain. I've seen a team set a global delay of 1 second and still get banned because they had 100 workers hitting the same domain simultaneously. The fix: per-domain token bucket. Each domain gets a bucket with capacity = 1 and refill rate = 1/crawl-delay. Workers acquire a token before fetching. This ensures you never exceed the delay, even with hundreds of workers. Also cache robots.txt with a TTL of 24 hours, but re-fetch if you get a 403 or 429.

// io.thecodeforge — System Design tutorial

// Per-domain token bucket rate limiter
class DomainRateLimiter {
    private buckets: Map<string, TokenBucket> = new Map()
    
    async acquire(domain: string): Promise<void> {
        let bucket = this.buckets.get(domain)
        if (!bucket) {
            const delay = await this.getCrawlDelay(domain)  // from robots.txt
            bucket = new TokenBucket(1, 1 / delay)  // capacity 1, refill 1 per delay seconds
            this.buckets.set(domain, bucket)
        }
        await bucket.consume()  // blocks until token available
    }
}

Deduplication: Never Fetch the Same URL Twice

Without deduplication, your crawler will fetch the same page thousands of times. The naive approach is a hash set of all seen URLs, but that doesn't scale — a billion URLs needs gigabytes of RAM. Use a Bloom filter. It's probabilistic: you might get false positives (skip a page you haven't seen) but never false negatives (never fetch a page twice). A Bloom filter with 1% false positive rate for 1 billion URLs needs about 1.2GB of memory. That's acceptable. For absolute correctness, combine with a Redis set for URLs that are 'seen' but not yet crawled. Check Bloom filter first, then Redis set. This reduces Redis memory usage by 99%.

// io.thecodeforge — System Design tutorial

// Bloom filter + Redis set for deduplication
class UrlDeduplicator {
    private bloom: BloomFilter  // initialized with expected count and false positive rate
    private redis: RedisClient
    
    async isDuplicate(url: string): Promise<boolean> {
        if (this.bloom.mightContain(url)) {
            // false positive possible, check Redis
            return await redis.sismember('crawled_urls', url)
        }
        return false
    }
    
    async markCrawled(url: string): Promise<void> {
        this.bloom.add(url)
        await redis.sadd('crawled_urls', url)
    }
}

Distributed Architecture: Crawling at Scale

A single machine can't crawl the entire web. You need a distributed system. The standard pattern: a master node manages the frontier and assigns work to worker nodes. Workers fetch pages, parse links, and send discovered URLs back to the master. Use a message queue like RabbitMQ or Kafka for communication. The master publishes URL batches to a 'to-crawl' queue; workers consume, crawl, and publish discovered URLs to a 'discovered' queue. The master consumes 'discovered' and adds to the frontier. This decouples workers and allows easy scaling. But watch out: if a worker crashes mid-crawl, you lose that page. Implement a 're-queue' mechanism: if a worker doesn't acknowledge within a timeout, the URL goes back to the queue.

// io.thecodeforge — System Design tutorial

// Master node pseudocode
async function masterLoop() {
    while (true) {
        // get next URL from frontier (respecting politeness)
        url = await frontier.nextUrl()
        // publish to RabbitMQ
        await channel.sendToQueue('to_crawl', Buffer.from(url))
    }
}

// Worker node pseudocode
async function workerLoop() {
    channel.consume('to_crawl', async (msg) => {
        const url = msg.content.toString()
        try {
            const html = await fetch(url)
            const links = parseLinks(html, url)
            // send discovered URLs to master
            for (const link of links) {
                await channel.sendToQueue('discovered', Buffer.from(link))
            }
            channel.ack(msg)
        } catch (err) {
            // re-queue on failure
            channel.nack(msg, false, true)
        }
    })
}

Handling JavaScript-Rendered Pages

Modern websites are SPAs that load content via JavaScript. A simple HTTP GET returns an empty shell. You need a headless browser like Puppeteer or Playwright. But this is expensive: each page takes seconds and consumes 100MB+ of RAM. Never use a headless browser for every page. First, try a regular fetch. If the page contains no meaningful content (e.g., no text, only scripts), then fall back to a headless browser. Cache the rendered HTML for a TTL. Also, use a pool of browsers to reuse contexts. I've seen a team spin up a new browser per page — they ran out of file descriptors in 10 minutes.

// io.thecodeforge — System Design tutorial

// Fallback to headless browser
async function fetchWithFallback(url: string): Promise<string> {
    const response = await fetch(url)
    const html = await response.text()
    if (hasMeaningfulContent(html)) {
        return html
    }
    // fallback to headless
    const browser = await pool.acquire()
    try {
        const page = await browser.newPage()
        await page.goto(url, { waitUntil: 'networkidle' })
        const content = await page.content()
        return content
    } finally {
        await page.close()
        pool.release(browser)
    }
}

Error Handling and Retries

The web is unreliable. You'll get timeouts, 500s, DNS failures, and connection resets. Your crawler must handle these gracefully. Implement exponential backoff with jitter. Start with 1 second, double each retry, cap at 60 seconds, and add random jitter to avoid thundering herd. Max retries: 3 for transient errors, 0 for 4xx (client errors). For 429 (rate limit), respect Retry-After header. I've seen a crawler retry a 404 five times — wasted bandwidth and filled logs. Also, log every error with context: URL, status code, response headers, and stack trace. This is invaluable for debugging.

// io.thecodeforge — System Design tutorial

async function fetchWithRetry(url: string, maxRetries = 3): Promise<Response> {
    for (let attempt = 0; attempt <= maxRetries; attempt++) {
        try {
            const response = await fetch(url)
            if (response.status === 429) {
                const retryAfter = response.headers.get('Retry-After') || '60'
                await sleep(parseInt(retryAfter) * 1000)
                continue
            }
            if (response.status >= 500 || response.status === 0) {
                // transient
                const delay = Math.min(1000 * Math.pow(2, attempt) + Math.random() * 1000, 60000)
                await sleep(delay)
                continue
            }
            return response
        } catch (err) {
            if (attempt === maxRetries) throw err
            const delay = Math.min(1000 * Math.pow(2, attempt) + Math.random() * 1000, 60000)
            await sleep(delay)
        }
    }
}

Storage: Where to Put All That Data

You need to store the crawled pages for indexing or analysis. The naive approach is to dump raw HTML into files. That doesn't scale. Use a distributed storage system like HDFS or S3. Partition by domain or date. Store metadata (URL, crawl timestamp, response headers, status code) in a database like Cassandra or DynamoDB for fast lookups. Raw HTML goes to blob storage. For small-scale (millions of pages), a single PostgreSQL database with text compression works. But watch out: storing HTML in a relational database kills performance. Use TOAST or separate blob storage. I've seen a team store 10TB of HTML in MySQL — the database became unusable.

// io.thecodeforge — System Design tutorial

// Store metadata in Cassandra, raw HTML in S3
async function storePage(url: string, html: string, headers: Headers, status: number) {
    const key = hashUrl(url)
    // metadata
    await cassandra.execute(
        'INSERT INTO pages (url_hash, url, crawl_time, status, content_type, s3_key) VALUES (?, ?, ?, ?, ?, ?)',
        [key, url, new Date(), status, headers.get('content-type'), `pages/${key}.html`]
    )
    // raw HTML to S3
    await s3.putObject({
        Bucket: 'crawled-pages',
        Key: `pages/${key}.html`,
        Body: html,
        ContentType: 'text/html'
    }).promise()
}

Monitoring and Observability

A crawler runs unattended for days. You need monitoring. Track: pages crawled per second, error rate by status code, queue depth, memory usage, and politeness violations. Alert on: error rate > 5%, queue depth > 100k, memory > 80%, or any 429 spike. Log every request with URL, status, duration, and worker ID. Use structured logging (JSON) and ship to Elasticsearch. I've seen a team not monitor queue depth — the master crashed, workers finished their queue and sat idle for 8 hours before anyone noticed. Also, set up a health endpoint that reports the crawler's state: frontier size, last crawl time per domain, and error counts.

// io.thecodeforge — System Design tutorial

// Health endpoint example (Express.js)
app.get('/health', async (req, res) => {
    const stats = {
        frontierSize: await redis.zcard('domains'),
        totalCrawled: await redis.get('stats:crawled') || 0,
        errorRate: await calculateErrorRate(),
        queueDepth: await channel.checkQueue('to_crawl'),
        memoryUsage: process.memoryUsage().heapUsed / 1024 / 1024
    }
    res.json(stats)
})