A Python parser that compares prices for the same products in 3 open online shops in one table

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

This parser puts the prices of the same products from up to three open online shops side by side: one row per product, with each shop's price, old price, stock and link, and the date of the check. The three shops come with Premium; Basic and Standard read one shop, as a start or a test. Products are matched by a code the shops show, such as a barcode or maker's part number, or by the links you list for each product: names alone differ too much to match safely. Send the sites, a product page of each saved from your browser and your product list. You get Python code, a settings file and a README and run it on your computer. It reads open pages at a polite pace, never logs in or gets around captchas or protection, and does not read Amazon or Google. You decide whether the shops' terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• One row per productEach shop's price, stock and link side by side
• Matched by code or your linksBarcode, maker's part number, or links you list
• Date of the checkIn every row, so runs can be compared
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   ShopsYou give the shops, saved product pages and your product list
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check a few rows in each shop
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The three shops (or one, for Basic and Standard) and a product page from each, saved from your browser (File > Save page as). Your product list with the code the shops show, such as a barcode, or a table with each product's link in every shop. The fields you need per shop, e.g. price, old price, in stock, link. Roughly how many products. CSV, Excel or JSON.

We will ask you:

• Which shops?
• How to match products
• The fields per shop
• How many products roughly?

Compare packages

Questions and answers

Did not find your answer? Ask us