A website parser in Python that collects public data from pages into CSV or Excel: prices, catalogues, listings

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

A parser collects data you can see on public web pages and saves it as a table: prices and stock of products, a supplier's catalogue for your shop, listings, job posts, events or articles. Tell us the site and the pages, list the fields you need (name, price, link, picture and so on) and send one or two of those pages saved from your browser (File > Save page as) or screenshots of them. We write the parser in Python by the real structure of those pages, and it saves the results as CSV, Excel or JSON. You get an archive with the code, a settings file with the addresses and the pause between pages, and a README that explains how to install Python and run it. You run the parser on your own computer, so the data goes straight to you; we do not collect it for you or keep it. It reads only open pages, at a polite pace, and never logs in, gets around captchas or other protection, or collects personal data such as people's names, emails or phone numbers. When a site changes its layout, the parser may need an update. Whether you may collect and use a site's data under its terms is your responsibility.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Your fields, your tableColumns in the order you need
• Built on the real pagesWritten by the structure of the pages you send
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python
• A plain guideWhat to install and how to run it, step by step

How it works

1. 1
   PagesYou give the site, the fields and one or two saved pages
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the first rows of the table
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the exact pages: a link to a list and to an item page, and roughly how many pages there are. The fields you need, in the order of your table, with an example of each. One or two of those pages saved from your browser as HTML (File > Save page as) or screenshots, so the parser follows their real structure. The results file you need (CSV, Excel or JSON) and where you will run it.

We will ask you:

• Which site and which pages?
• The fields you need
• How many pages roughly?
• The results file
• Anything else it should do

Compare packages

Questions and answers

Did not find your answer? Ask us