Scrapy: Production Web Scraping Framework in Python

Web scraping is a cornerstone of data-driven applications, from price monitoring to lead generation. While you can scrape with libraries like Requests and BeautifulSoup, production-grade scraping demands a framework that handles concurrency, retries, middleware, and data pipelines. Enter Scrapy: an asynchronous framework built for speed and reliability.

Scrapy is not just a library; it's a complete ecosystem. It provides a spider architecture, item pipelines, exporters, and a built-in shell for testing. With Scrapy, you can scrape thousands of pages per minute while respecting website policies and avoiding bans.

In this tutorial, you'll learn how to build a production-ready Scrapy spider from scratch. We'll cover real-world challenges: handling JavaScript-rendered content, rotating user agents and proxies, managing cookies, and debugging common failures. By the end, you'll be able to deploy a spider that runs reliably in a production environment.

We'll use a fictional e-commerce site as our target. The code examples are tested and ready to adapt. Let's dive into the world of production web scraping with Scrapy.

1. Setting Up Scrapy Project

Before writing spiders, you need to create a Scrapy project. This organizes your code into a standard structure with settings, spiders, items, and pipelines.

First, install Scrapy using pip. It's recommended to use a virtual environment to avoid conflicts.

``bash pip install scrapy ``

Create a new project:

``bash scrapy startproject myproject cd myproject ``

Now, generate a spider template:

``bash scrapy genspider example example.com ``

This creates a basic spider in spiders/example.py. You can edit it to define your scraping logic.

Production Tip: Always use a dedicated virtual environment and pin your dependencies in a requirements.txt file. This ensures reproducibility across environments.

# Create a virtual environment
python -m venv scrapy_env
source scrapy_env/bin/activate  # On Windows: scrapy_env\Scripts\activate

# Install Scrapy
pip install scrapy

# Start a project
scrapy startproject myproject
cd myproject

# Generate a spider
scrapy genspider example example.com

2. Writing Your First Spider

A spider is a class that defines how to scrape a website. It must subclass scrapy.Spider and define the name, start_urls, and parse method.

Here's a simple spider that scrapes quotes from a demo site:

```python import scrapy

class QuotesSpider(scrapy.Spider): name = "quotes" start_urls = [ 'http://quotes.toscrape.com/page/1/', ]

def parse(self, response): for quote in response.css('div.quote'): yield { 'text': quote.css('span.text::text').get(), 'author': quote.css('small.author::text').get(), 'tags': quote.css('div.tags a.tag::text').getall(), } # Follow pagination link next_page = response.css('li.next a::attr(href)').get() if next_page is not None: yield response.follow(next_page, callback=self.parse) ```

To run the spider:

``bash scrapy crawl quotes -o quotes.json ``

This outputs scraped data to quotes.json.

Production Tip: Use scrapy crawl quotes -o quotes.jl for JSON Lines format, which is append-friendly and easier to process in pipelines.

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'http://quotes.toscrape.com/page/1/',
    ]

    def parse(self, response):
        for quote in response.css('div.quote'):
            yield {
                'text': quote.css('span.text::text').get(),
                'author': quote.css('small.author::text').get(),
                'tags': quote.css('div.tags a.tag::text').getall(),
            }
        
        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

3. Using Item Pipelines for Data Processing

Item pipelines are used to process scraped items after they are yielded by spiders. Common tasks include cleaning, validating, deduplicating, and storing data.

Define an item class in items.py:

```python import scrapy

class QuoteItem(scrapy.Item): text = scrapy.Field() author = scrapy.Field() tags = scrapy.Field() ```

Then create a pipeline in pipelines.py:

```python class CleanDataPipeline: def process_item(self, item, spider): # Strip whitespace from text fields item['text'] = item['text'].strip() item['author'] = item['author'].strip() # Convert tags to lowercase item['tags'] = [tag.strip().lower() for tag in item['tags']] return item

class DuplicatesPipeline: def __init__(self): self.seen = set()

def process_item(self, item, spider): if item['text'] in self.seen: raise DropItem(f"Duplicate item found: {item}") else: self.seen.add(item['text']) return item ```

Enable pipelines in settings.py:

``python ITEM_PIPELINES = { 'myproject.pipelines.CleanDataPipeline': 300, 'myproject.pipelines.DuplicatesPipeline': 400, } ``

The numbers define the order (lower numbers run first).

Production Tip: Use a database pipeline to store items directly into PostgreSQL or MongoDB. For high throughput, consider using a message queue like RabbitMQ between Scrapy and your storage.

from scrapy.exceptions import DropItem

class CleanDataPipeline:
    def process_item(self, item, spider):
        item['text'] = item['text'].strip()
        item['author'] = item['author'].strip()
        item['tags'] = [tag.strip().lower() for tag in item['tags']]
        return item

class DuplicatesPipeline:
    def __init__(self):
        self.seen = set()

    def process_item(self, item, spider):
        if item['text'] in self.seen:
            raise DropItem(f"Duplicate item found: {item}")
        else:
            self.seen.add(item['text'])
            return item

4. Handling JavaScript-Rendered Content with Splash

Many modern websites rely on JavaScript to render content. Scrapy's default HTTP client cannot execute JavaScript. To handle this, you can use Scrapy-Splash, which integrates the Splash JavaScript rendering service.

First, install Scrapy-Splash:

``bash pip install scrapy-splash ``

Then configure your settings:

```python SPLASH_URL = 'http://localhost:8050' # Splash instance

DOWNLOADER_MIDDLEWARES = { 'scrapy_splash.SplashCookiesMiddleware': 723, 'scrapy_splash.SplashMiddleware': 725, 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware': 810, }

SPIDER_MIDDLEWARES = { 'scrapy_splash.SplashDeduplicateArgsMiddleware': 100, }

DUPEFILTER_CLASS = 'scrapy_splash.SplashAwareDupeFilter' ```

Now you can use SplashRequest in your spider:

```python import scrapy from scrapy_splash import SplashRequest

class JavaScriptSpider(scrapy.Spider): name = "js_spider" start_urls = ['https://example.com/js-page']

def start_requests(self): for url in self.start_urls: yield SplashRequest(url, self.parse, args={'wait': 2})

def parse(self, response): # Now response contains the rendered HTML yield { 'title': response.css('h1::text').get(), } ```

The wait argument tells Splash to wait for 2 seconds after page load to allow JavaScript to execute.

Production Tip: Run Splash in a Docker container. Use a pool of Splash instances behind a load balancer for high traffic.

import scrapy
from scrapy_splash import SplashRequest

class JavaScriptSpider(scrapy.Spider):
    name = "js_spider"
    start_urls = ['https://example.com/js-page']

    def start_requests(self):
        for url in self.start_urls:
            yield SplashRequest(url, self.parse, args={'wait': 2})

    def parse(self, response):
        yield {
            'title': response.css('h1::text').get(),
        }

5. Avoiding Bans: Rotating User Agents and Proxies

Websites often block scrapers that send too many requests from the same IP with the same User-Agent. To avoid bans, you need to rotate user agents and proxies.

User-Agent Rotation:

Install scrapy-user-agents or write a custom middleware:

```python # middlewares.py import random from scrapy import signals

class RandomUserAgentMiddleware: def __init__(self, user_agents): self.user_agents = user_agents

@classmethod def from_crawler(cls, crawler): return cls(user_agents=crawler.settings.get('USER_AGENTS_LIST'))

def process_request(self, request, spider): request.headers['User-Agent'] = random.choice(self.user_agents) ```

In settings.py:

```python USER_AGENTS_LIST = [ 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Safari/605.1.15', # Add more real user agents ]

DOWNLOADER_MIDDLEWARES = { 'myproject.middlewares.RandomUserAgentMiddleware': 400, } ```

Proxy Rotation:

Use scrapy-proxy-pool or a custom middleware. For example, with a list of proxies:

```python class RandomProxyMiddleware: def __init__(self, proxies): self.proxies = proxies

@classmethod def from_crawler(cls, crawler): return cls(proxies=crawler.settings.get('PROXY_LIST'))

def process_request(self, request, spider): request.meta['proxy'] = random.choice(self.proxies) ```

Production Tip: Use a proxy service like Luminati or Smartproxy that provides rotating residential IPs. Avoid free proxies as they are unreliable and often blocked.

import random

class RandomUserAgentMiddleware:
    def __init__(self, user_agents):
        self.user_agents = user_agents

    @classmethod
    def from_crawler(cls, crawler):
        return cls(user_agents=crawler.settings.get('USER_AGENTS_LIST'))

    def process_request(self, request, spider):
        request.headers['User-Agent'] = random.choice(self.user_agents)

class RandomProxyMiddleware:
    def __init__(self, proxies):
        self.proxies = proxies

    @classmethod
    def from_crawler(cls, crawler):
        return cls(proxies=crawler.settings.get('PROXY_LIST'))

    def process_request(self, request, spider):
        request.meta['proxy'] = random.choice(self.proxies)

6. Scaling Scrapy with Settings and Extensions

Scrapy provides numerous settings to control concurrency, delays, and resource usage. Proper tuning is essential for production.

Key settings:

• CONCURRENT_REQUESTS: Number of simultaneous requests (default 16). Increase for faster scraping, but beware of bans.
• DOWNLOAD_DELAY: Delay between consecutive requests (seconds). Use to throttle requests.
• AUTOTHROTTLE_ENABLED: Automatically adjusts delay based on server response times.
• COOKIES_ENABLED: Enable if the site uses cookies for sessions.
• RETRY_ENABLED: Retry failed requests.
• DOWNLOAD_TIMEOUT: Timeout for requests.

Example configuration:

``python # settings.py CONCURRENT_REQUESTS = 32 DOWNLOAD_DELAY = 1 AUTOTHROTTLE_ENABLED = True AUTOTHROTTLE_START_DELAY = 5 AUTOTHROTTLE_MAX_DELAY = 60 COOKIES_ENABLED = False RETRY_ENABLED = True RETRY_TIMES = 3 DOWNLOAD_TIMEOUT = 30 ``

Extensions: Scrapy has built-in extensions for logging, stats collection, and memory usage. You can also write custom extensions.

For example, to log memory usage:

```python # extensions.py from scrapy import signals import psutil

class MemoryUsageExtension: def __init__(self, crawler): self.crawler = crawler

@classmethod def from_crawler(cls, crawler): ext = cls(crawler) crawler.signals.connect(ext.spider_closed, signal=signals.spider_closed) return ext

def spider_closed(self, spider): mem = psutil.Process().memory_info().rss / (1024 ** 2) spider.logger.info(f"Memory usage: {mem:.2f} MB") ```

Enable in settings:

``python EXTENSIONS = { 'myproject.extensions.MemoryUsageExtension': 500, } ``

Production Tip: Use scrapy crawl spider -s LOG_LEVEL=INFO to control logging verbosity. In production, set LOG_FILE to write logs to a file for debugging.

# settings.py
BOT_NAME = 'myproject'

SPIDER_MODULES = ['myproject.spiders']
NEWSPIDER_MODULE = 'myproject.spiders'

ROBOTSTXT_OBEY = True

CONCURRENT_REQUESTS = 32
DOWNLOAD_DELAY = 1
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5
AUTOTHROTTLE_MAX_DELAY = 60
COOKIES_ENABLED = False
RETRY_ENABLED = True
RETRY_TIMES = 3
DOWNLOAD_TIMEOUT = 30

ITEM_PIPELINES = {
    'myproject.pipelines.CleanDataPipeline': 300,
    'myproject.pipelines.DuplicatesPipeline': 400,
}

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RandomUserAgentMiddleware': 400,
    'myproject.middlewares.RandomProxyMiddleware': 500,
}

7. Deploying Scrapy Spiders to Production

Deploying Scrapy spiders involves running them on a server or cloud platform. Common approaches include:

1. Scrapyd: A service to run Scrapy spiders remotely. Install with pip install scrapyd and start the server. Then deploy your project using scrapyd-deploy.
2. Docker: Containerize your spider and run it on any platform (AWS, GCP, Azure).
3. Serverless: Use AWS Lambda with scrapy-lambda or similar.

Docker example:

Create a Dockerfile:

```dockerfile FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt . RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["scrapy", "crawl", "quotes"] ```

Build and run:

``bash docker build -t my-scrapy-spider . docker run my-scrapy-spider ``

Scheduling: Use cron jobs or a scheduler like Apache Airflow to run spiders periodically.

Monitoring: Integrate with logging services (e.g., ELK stack) and set up alerts for failures.

Production Tip: Use a secrets manager (e.g., AWS Secrets Manager) to store API keys, proxy credentials, and database passwords. Never hardcode them.

FROM python:3.9-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

CMD ["scrapy", "crawl", "quotes"]