A Python parser that saves pictures from an open catalogue to a folder, named by product code, with a table

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

When you move your own catalogue to a new shop, or a supplier lets you use its product photos, this parser saves the pictures from open catalogue pages into a folder and names each file by the product code, with a table that links code, product name, picture link and file name. Send the site, a category and a product page, one or two pages saved from your browser and how to name the files. You get Python code, a settings file and a README and run it on your computer, so the files go straight to you. Save only pictures you may use: your own, or ones the owner allows. Files are saved as the site gives them. It reads open pages at a polite pace and never logs in or gets around captchas or protection. You decide whether the site's terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Files named your wayBy product code, with _1, _2, _3 for extra photos
• A table to go with themCode, product name, picture link and file name
• The largest versionWhen the page links to a bigger picture, it takes that one
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the catalogue, a saved product page and the file names
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and look through the first folder of pictures
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the exact pages: a link to a category and to one product page, and roughly how many products and pictures there are. How to name the files, e.g. by product code with _1, _2. The table fields you need besides the file name. One or two of those pages saved from your browser (File > Save page as). And your confirmation that the pictures are yours or that the owner lets you use them.

We will ask you:

• Which site and pages?
• How to name the files
• How many pictures roughly?
• Whose pictures are they?

Compare packages

Questions and answers

Did not find your answer? Ask us