A Python parser that collects open product reviews into a table: text, rating and date only, no names

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

To see what buyers like and dislike about your products or a competitor's, this parser collects open reviews into a table: the product and its link, and for each review only the text, the rating and the date. Reviewers' names, nicknames, photos and profile links are left out, so the table holds no personal data. Send the shop's site, a product page with reviews and one or two pages saved from your browser; when reviews open with a «show more» button, the parser follows it. You get Python code, a settings file and a README and run it on your computer. It does not read Amazon, Google, LinkedIn or social networks, never logs in or gets around captchas or protection, and keeps a polite pace. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Three facts per reviewText, rating and date, nothing about the person
• Linked to the productEach review row carries the product name and link
• All review pagesFollows «show more» and the review pages
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the shop, the products and a saved page with reviews
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the review count of one product
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The shop's site and the exact pages: a list of product links or a category link, and roughly how many products and reviews there are. The product fields you need besides the review text, rating and date, e.g. product name, code, link, average rating. One or two pages with reviews saved from your browser (File > Save page as) or screenshots. CSV or Excel.

We will ask you:

• Which shop and products?
• Product fields
• How many reviews roughly?
• Only some reviews?

Compare packages

Questions and answers

Did not find your answer? Ask us