A Python parser that collects a competitor's public prices into a table: price, old price, stock and link

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

This parser reads a competitor's public product pages and puts their prices into a table you can set against your own: product name, product code, price, old price, discount, in stock and link, with the date of the check. Give us the site, links to a category and to a product page, the fields in the order of your table and one or two of those pages saved from your browser. We write the code in Python by their real structure, with a settings file and a README. You run it on your computer whenever you want fresh prices, and the table comes straight to you. It reads only open pages at a polite pace and never logs in, gets around captchas or protection, or collects personal data. You decide whether the site's terms allow it, and a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Price columnsPrice, old price and discount as separate numbers
• Codes for matchingThe competitor's code or barcode, when the page shows it
• Date of each checkEvery row carries the date it was collected
• Polite paceA pause between pages that you set
• Runs on your computerWindows, macOS or Linux; run it again for fresh prices

How it works

1. 1
   PagesYou give the competitor's site, a category and a saved product page
2. 2
   ParserWe write the code by those pages and the guide to running it
3. 3
   RunYou run it and compare the first rows with the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The competitor's site and the exact pages: a link to a category and to a product page, and roughly how many products there are. The fields you need in the order of your table, e.g. name, code, price, old price, in stock, link. One or two of those pages saved from your browser as HTML (File > Save page as) or screenshots. CSV or Excel, and where you will run it.

We will ask you:

• Which competitor site and pages?
• The fields you need
• How many products roughly?
• How often will you run it?

Compare packages

Questions and answers

Did not find your answer? Ask us