Create a Scrapy Project

Scrapy has, an efficient command-line tool, also called the ‘Scrapy tool’. Commands accept a different set of arguments and options based on their purpose. To write the Spider code, we begin by creating, a Scrapy project, by executing the following command, at the terminal –

scrapy startproject <project_name>

Output:

This should create a folder, in your current directory. It contains a ‘scrapy.cfg’, which is a configuration file, of the project. The folder structure is as shown below:

The scrapy.cfg, is a project configuration file. The folder, that contains this file, is the root directory. The folder structure, of folder, created is as follows:

The folder, contains items.py,middlerwares.py and other settings files, along with the ‘spiders’ folder. The crawling code will be written, in a spider python file. We will alter, ‘items.py’ file, to mention, our data items, to be extracted. Keep the contents of ‘items.py’, as they are currently.

In this article, we will scrape Quotes data using scrapy items, from the webpage https://quotes.toscrape.com/tag/reading/. The main objective of scraping, is to prepare structured data, from unstructured resources. Scrapy Items are wrappers around, the dictionary data structures. Code can be written, such that, the extracted data is returned, as Item objects, in the format of “key-value” pairs.  Using Scrapy Items is beneficial when –

• As the scraped data volume increases, they become irregular to handle.
• As your data gets complex, it is vulnerable to typos, and, at times may return faulty data.
• Formatting of data scraped, is easier, as Item objects, can be further passed to Item Pipelines.
• Cleansing the data, is easy, if we scrape the data, as Items.
• Validating data, handling missing data, is easier with Scrapy Items.

Via the Item adapter library, Scrapy supports various Item Types. One can choose, the Item type, they want. Following, are the Item Types supported:

• Dictionaries – Items can be written in form of dictionary objects. They are convenient to use.
• Item objects – They provide dictionary like API, wherein we need to declare, fields for the Item, needed. It consists of key-value pair, of Field objects used, while declaring the Item class. In this tutorial, we are using Item objects.
• Dataclass objects – They are used, when you need to store, the scraped values, in JSON or CSV files. Here we need to define, the datatype of each field, needed.
• attr.s – The attr.s allows, defining item classes, with field names, so that scraped data, can be imported, to different file formats. They work similar to Dataclass objects only that the attr package needs to be installed.