Group Information Retrieval Practicals

Q7: Build a web crawler to collect product information and links from an e-commerce website. Respect the website's terms, robots.txt, rate limits, and access controls; demonstrate with a site you own or have permission to crawl.

Web Crawler for E-commerce Product Information

Solution and implementation for Q7 from Information Retrieval (ir).

7_focused_product_crawler.py Download
"""Read product names and links from the Books to Scrape demo website."""

import requests
from bs4 import BeautifulSoup
from urllib.parse import urljoin


url = "https://books.toscrape.com/"
response = requests.get(url, timeout=15)
response.raise_for_status()
soup = BeautifulSoup(response.text, "html.parser")

for product in soup.find_all("article", class_="product_pod"):
    name = product.h3.a["title"]
    price = product.find("p", class_="price_color").get_text(strip=True)
    link = urljoin(url, product.h3.a["href"])
    print("Product:", name)
    print("Price:", price)
    print("Link:", link)
    print("-" * 40)

Other Questions in Information Retrieval

See All Available Questions
Download