How to pass a user defined argument in scrapy spider

In Scrapy, passing user-defined parameters can be achieved through multiple methods. The most common approach is to pass parameters via the command line when starting a spider. Alternatively, parameters can be passed within the code by overriding the __init__ method. Below, I will detail both methods.

Method 1: Passing Parameters via Command Line

When using the command line to start a Scrapy spider, you can use the -a option to pass parameters. These parameters will be passed to the spider's constructor and can be utilized within the spider.

Example:

Assume you have a spider named MySpider that needs to scrape data for different categories based on a user-provided category parameter.

First, in the spider code, you can access this parameter as follows:

python
import scrapy

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def __init__(self, category=None, *args, **kwargs):
        super(MySpider, self).__init__(*args, **kwargs)
        self.category = category

    def start_requests(self):
        url = f'http://example.com/{self.category}'
        yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        # This is where your parsing logic resides
        pass

Next, when launching the spider via the command line, you can pass the parameters as follows:

bash
scrapy crawl my_spider -a category=books

This will cause the spider to construct the request URL based on the provided category parameter value books.

Method 2: Setting Parameters in Code

If you prefer to set parameters directly within the code rather than via the command line, you can pass parameters directly to the __init__ method when creating a spider instance. This is typically used when dynamically creating a spider and passing parameters within a script.

Example:

python
from scrapy.crawler import CrawlerProcess
from myspiders import MySpider

process = CrawlerProcess({
    'USER_AGENT': 'Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.1)'
})

category = 'electronics'
spider = MySpider(category=category)
process.crawl(spider)
process.start()

Here, the category parameter is directly passed when instantiating the MySpider object.

Summary

With both methods, you can flexibly pass custom parameters to Scrapy spiders, enabling dynamic adjustment of the spider's behavior based on varying requirements. This is particularly valuable for handling crawling tasks that must adapt to user input or other changing conditions.