A Python parser that checks whether products are in stock on an open site and saves each status with the date

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Each time you run it, this parser checks which products are in stock on an open site, such as your supplier's or a shop whose goods you resell, and saves a table: name, code, status as the page writes it, quantity left when shown, delivery time, price, link, and the date and time of the check. Keep the file from each run and you see what came back and what ran out. Send the site, links to a category and a product page, the statuses the site uses and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. It reads only open pages at a polite pace and never logs in, gets around captchas or protection, or collects personal data. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Status as a columnThe site's words, or your own labels such as yes / no / on order
• Quantity when shownWhen the page shows how many are left
• Dated rowsDate and time of the check in every row
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the site, the products and the statuses it uses
2. 2
   ParserWe write the code and the guide to running it
3. 3
   First checkYou run it and compare a few statuses with the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the exact pages: a link to a category and to a product page, or your list of product links, and roughly how many products. The stock statuses the site shows, written as they appear. The fields you need besides the status, e.g. code, price, delivery time. One or two pages saved from your browser (File > Save page as) or screenshots, ideally one in stock and one out of stock. How often you plan to run it.

We will ask you:

• Which site and products?
• The stock statuses
• Other fields
• How often will you check?

Compare packages

Questions and answers

Did not find your answer? Ask us