A Python parser that collects product specifications from open pages into columns for side-by-side comparison

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Shops filling their own product cards and buyers choosing between products use this parser to turn the spec tables of open product pages into columns: name, code, price, and each spec, such as size, weight, power, material or warranty, in its own column with the unit apart. Products list different specs, so you name the ones you need, up to the field limit of your package; missing ones stay as empty cells. Send the site, a category, two product pages saved from your browser and your list of specs. You get Python code, a settings file and a README and run it on your computer. Specs are copied as the page writes them; we do not check or complete them. It reads open pages at a polite pace and never logs in or gets around captchas or protection. You decide whether the site's terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• One spec, one columnReady to set products side by side
• Units aparte.g. 1200 and W in separate columns
• Your spec listOnly the specs you name, in your order
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the site, two saved product pages and your spec list
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the specs of two products
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the exact pages: a link to a category and to two product pages with different specs, and roughly how many products there are. Your list of specs in the order of your table, written as the site writes them, and whether units go in separate columns. Two pages saved from your browser (File > Save page as) or screenshots. CSV or Excel.

We will ask you:

• Which site and pages?
• Your spec list
• How many products roughly?
• Units

Compare packages

Questions and answers

Did not find your answer? Ask us