In this article, I'll be explaining the working of a Python web crawler whose purpose is to extract all the images from the given URLs. I created this Python web crawler to grab all images of all formats: .img .jpg .jpeg .png .webp .bmp. This crawler can also grab GIFs from the web pages and it is not limited to just one webpage or URL. You can crawl/scrap multiple webpages or URLs.
Scraping a webpage without permission is a violation of the website's terms of service. Make sure the webpage that you are trying to crawl either does not restrict scraping or you have appropriate permission to scrap that webpage.
Creating python environment and required packages
First you need setup virtual environment to isolate the current project. Check How to create python virtual environment article. Most of the libraries used for this code are Pyhton in-build libraries. These are some of the external packages that you have install manually:
pip install beautifulsoup4 tqdm
- In the script, Beautiful Soup 4 is used to extract data from HTML webpages. All that parsing, navigation, searching, and modification is done amazingly with the help of beautiful soup.
- Another library is tqdm which means progress in Arabic. As the name states, this library is used to represent the progress of operations running by the script.
Functioning of crawler
After running the script using python getimages.py command, user just has to input the URLs. Now if you have more than one URL, paste them one by one in the input field, make sure you separate URLs with commas otherwise it won't work.Check the below image for reference:
Now the beauty of this crawler is, It works for both relative and absolute links. nowadays, there are many websites that use relative links in their image tags. But it won't be an issue with the crawler as this crawler will automatically convert the relative link into an absolute link, which will be used to download images.
Scrapped images will be saved in the current working directory under a new directory by the name of python-image-crawler. This directory will be created automatically upon the initialization of this script.
Download Python crawler script
Script is available on my GitHub repository. Click on the button below to clone the repository:
Now there is a high probability that you will get 403 Forbidden error because website these use cloudflare, which will automatically block scrappers. You need to make sure that you have appropriate permission to crawl that website. Below are some of the links that won't restrict scraping. You can use their links to check the script.
Sample URLs:https://github.com/,https://dribbble.com/,https://www.amazon.in/gp/bestsellers/?ref_=nav_cs_bestsellers
