Inspecting the site

Inspecting the site

This is the most crucial part of web scraping to inspect and analyze various aspects of the website. This allows us to get familiar with the structure and components of the website. If you are using chrome, developers have a great option of Dev tools, such as inspecting the webpage and see the tags and elements associated with it....

Finding the target element or tag

Now move on to the actual site’s content and find our target, which in this case is the current price of the cryptocurrency. We can use the “Select the element from the page to inspect” option on the top left side of the inspect/ developer window. This allows us to see the elements/tags by clicking or hovering over them. This will make sure we are selecting the correct tag from the source of the webpage. Also, it becomes quite easy to find our target element as we can visually see the element....

Using cURL to access the website and store it in a file

Now we move on to the actual scrapping and accessing the website page from the terminal. cURL command is a great option for this as it is available by default in many Linux/Unix systems. We also have an alternative cURL called wget, but is not widely available in many systems. We can access a website URL using the following command....

Scraping the webpage file using grep and sed

We now have the source of the webpage from which we need the price of the cryptocurrency coins. We will start by using grep to tick out the span tag with the class no wrap and data-price-BTC, which’s value changes every time, so we will check it here. We won’t be able to scrap the content if we hard code the value, it may change every second, so just keep it till there....

Printing the price

We are almost done. We only need to print the price, as we have stored the price in the file, we need to store it in the variable to print it according to our needs. We will use a while loop until the file reaches the end of the file and extract the only line into a variable. Finally, we use the echo command to print the price....