Following Links
Till now, we have seen the code, to extract data, from a single webpage. Our final aim is to fetch, the Quote’s related data, from all the web pages. To do so, we need to make our spider, follow links, so that it can navigate, to the subsequent pages. The hyperlinks are usually defined, by writing <a> tags. The “href” attribute, of the <a> tags, indicates the link’s destination. We need to extract, the “href” attribute, to traverse, from one page to another. Let us study, how to implement the same –
- To traverse to the next page, check the CSS attribute of the “Next” hyperlink.
The CSS class attribute of “Next ->” hyperlink is “next”
We need to extract, the “href” attribute, of the <a> tag of HTML. The “href” attribute, denotes the URL of the page, where the link goes to. Hence, we need to fetch the same, and, join to our current path, for the spider to navigate, to further pages seamlessly. For the first page, the “href” value of <a> tag is, “/page/2”, which means, it links to the second page.
If you click, and, observe the “Next” link of the second webpage, it has a CSS attribute as “next”. For this page, the “href” value of <a> tag, is “/page/3” which means, it links to the third page, and so on.
The “href” attribute of “Next” link on page2, links to the 3rd webpage
Hence, the XPath expression, for the next page link, can be fetched writing expression as – further_page_url = response.xpath(‘//*[@class=”next”]/a/@href’).extract_first(). This will give us, value of “@href” , which is “/page/2” for the first page.
The URL above, is not sufficient, to make the spider crawl, to the next page. We need to form, an absolute URL, by merging the response object URL, with the above relative URL. To do so, we will use urljoin() method.
The Response object URL is “https://quotes.toscrape.com/”. To travel, to the next page, we need to join it, with the relative URL “/page/2”. The syntax, for the same is – complete_url_next_page = response.urljoin(further_page_url). This syntax, will give us, the complete path as, “https://quotes.toscrape.com/page/2/”. Similarly, for second page, it will modify, according to the webpage number, as “https://quotes.toscrape.com/page/3/” and so on.
The parse method, will now make a new request, using this ‘complete_url_next_page ‘ URL.
Hence, our final Request object, for navigating to the second page, and crawling it, will be – yield scrapy.Request(complete_url_next_page). The complete code of the spider will be as follows:
Python3
# Import the required libraries import scrapy # Spider class name class GfgSpilinkSpider(scrapy.Spider): # Name of the spider name = 'gfg_spilink' # The domain to be scraped allowed_domains = ['quotes.toscrape.com'] # The URLs to be scraped from the domain start_urls = ['http://quotes.toscrape.com/'] # Default callback method def parse(self, response): quotes = response.xpath('//*[@class="quote"]') for quote in quotes: # XPath expression to fetch # text of the Quote title title = quote.xpath('.//*[@class="text"]/text()').extract_first() # XPath expression to fetch # author of the Quote authors = quote.xpath('.//*[@itemprop="author"]/text()').extract() tags = quote.xpath('.//*[@itemprop="keywords"]/@content').extract() yield {"Quote Text ": title, "Authors ": authors, "Tags ": tags} # Check CSS attribute of the "Next" # hyperlink and extract its "href" value further_page_url = response.xpath( '//*[@class="next"]/a/@href').extract_first() # Append the "href" value, to the current page, # to form a complete URL, of next page complete_url_next_page = response.urljoin(further_page_url) # Make the spider crawl, to the next page, # and extract the same data # A new Request with the URL is made yield scrapy.Request(complete_url_next_page) |
Execute the Spider, at the terminal, by using the command ‘crawl’. The syntax is as follows – scrapy crawl spider_name. Hence, we can run our spider as – scrapy crawl gfg_spilink. It will crawl, the entire website, by following links, and yield the Quotes data. The output is as seen below –
The Spider outputs Quotes from webpage 1 , 2 and rest of them
If we check, the Spider output statistics, we can see that the Spider has crawled, over ten webpages, by following the links. Also, the number of Quotes is close to 100.
The Spider statistics, at the terminal, indicating the number of pages crawled
We can collect data, in any file format, for storage or analysis. To collect the same, in a JSON file, we can mention the filename, in the ‘crawl’, syntax as follows:
scrapy crawl gfg_spilink -o spiderlinks.json
The above command will collect the entire scraped Quotes data, in a JSON file “spiderlinks.json”. The file contents are as seen below:
All Quotes are collected in JSON file
How To Follow Links With Python Scrapy ?
In this article, we will use Scrapy, for scraping data, presenting on linked webpages, and, collecting the same. We will scrape data from the website ‘https://quotes.toscrape.com/’.