A Python parser that turns a supplier's open catalogue into an import file for your shop, with photo links

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

When a supplier shows its range on an open site but sends no file, this parser copies the catalogue into a table your shop can import: name, code, price, description, specs, category and photo links, and it saves the photos to a folder. Send the supplier's site, links to a category and a product page, one or two pages saved from your browser and your shop's import template, for example a Shopify or WooCommerce CSV, so the columns follow it. You get Python code with a settings file and a README and run it on your computer. Use only photos and texts the supplier lets you use. It reads open pages at a polite pace and never logs in, gets around captchas or protection, or collects personal data. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Columns for your importIn the order and with the names of your shop's import file
• Photos ready to loadPhoto links in a column and files named by product code
• Prices as shownThe supplier's price; your markup you add in the table
• Polite paceA pause between pages, set in the settings
• A plain guideWhat to install and how to run it, step by step

How it works

1. 1
   CatalogueYou give the supplier's pages and your import template
2. 2
   ParserWe write the code and the guide to running it
3. 3
   Test importYou run it and import the first rows into your shop
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The supplier's site: a link to a category and to a product page, and roughly how many products there are. Your shop's import template or a sample export of a few of your products, so the columns match. The fields you need, such as name, code, price, description, specs, category, photos. One or two supplier pages saved from your browser (File > Save page as). How to name photo files. And your confirmation that the supplier lets you use its photos and texts.

We will ask you:

• Which supplier site and pages?
• Your shop and its import file
• How many products roughly?
• Photos

Compare packages

Questions and answers

Did not find your answer? Ask us