A Python parser for an open company directory: name, category, city, address and website, no personal contacts

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

For market research or a list of possible partners, this parser copies an open business directory into a table: company name, category, city, address, website link, short description, opening hours and the link to the company's page in the directory. It collects no personal contacts: no names of owners, directors or staff, no emails and no phone numbers, so the table is not a mailing or calling list. Send the directory site, a category page, a company page and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. It reads open pages at a polite pace, never logs in or gets around captchas or protection, and does not read Google Maps, LinkedIn or social networks. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Companies, not peopleNo names, emails or phone numbers
• Category and cityIn separate columns for filters
• Website linkSo you can visit each company's own site
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   DirectoryYou give the directory, a category and a saved company page
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the first companies against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The directory site and the exact pages: a link to a category (and city, if the site has that filter) and to one company page, and roughly how many companies there are. The fields in the order of your table, e.g. name, category, city, address, website, description. One or two of those pages saved from your browser (File > Save page as) or screenshots. CSV or Excel.

We will ask you:

• Which directory and pages?
• Categories and cities
• The fields you need
• How many companies roughly?

Compare packages

Questions and answers

Did not find your answer? Ask us