A Python parser that collects open book prices into a table: title, author, ISBN, format, price and stock

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Bookshops and second-hand sellers use this parser to see what open online shops charge: title, author as printed on the book, ISBN, publisher, year, format (hardcover, paperback, e-book), price, old price, in stock and link. With the ISBN in each row you can match books with your own list in Excel. Send the shop's site, a section or search page, one book page and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer whenever you reprice. It collects no data about readers or private sellers. It reads only open pages at a polite pace, never logs in or gets around captchas or protection, and does not read Amazon or Google. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• ISBN for matchingIn its own column, ready for a lookup in Excel
• Price and old priceAs numbers, with stock next to them
• Formats apartHardcover, paperback and e-book as separate rows
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the shop, a section and one saved book page
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check a few prices against the shop
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The bookshop's site and the exact pages: a link to a section or a search and to one book page, and roughly how many books there are. The fields in the order of your table, e.g. title, author, ISBN, publisher, year, format, price, old price, in stock, link. One or two of those pages saved from your browser (File > Save page as) or screenshots. Your own ISBN list, if you want to read only those books.

We will ask you:

• Which shop and pages?
• The fields you need
• How many books roughly?
• Anything else it should do

Compare packages

Questions and answers

Did not find your answer? Ask us