A Python parser for open car listings: make, year, mileage, price and city in a table, no seller contacts

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Dealers and car buyers use this parser to see the market in one table: make, year, mileage, engine, gearbox, body, fuel, colour, price, city, date posted, link and photo links from open car listings. Set up the search on the site (make, years, price range) and send that link, one listing and one or two pages saved from your browser. We write the code in Python by those pages; you get it with a settings file and a README and run it yourself as often as you need. Seller data stays out: no names, emails or phone numbers, and no number plates or VIN. The parser reads open pages at a polite pace and never logs in or gets around captchas or protection. Whether the site's terms allow it is your decision, and a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• The market in one tableSort by year, mileage or price in Excel
• Your search linkMake, years and price range set on the site
• No seller dataNo names, phones, plates or VIN
• Mileage as a numberReady for filters and averages
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   SearchYou send a search link and one saved listing
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check a few cars against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The car site and a link to a search with your filters, one listing page and roughly how many cars the search shows. The fields you need in the order of your table, e.g. make, year, mileage, engine, gearbox, price, city, link. One or two of those pages saved from your browser (File > Save page as) or screenshots. CSV or Excel, and how often you will run it.

We will ask you:

• Which site and search?
• The fields you need
• How many cars roughly?
• Anything else it should do

Compare packages

Questions and answers

Did not find your answer? Ask us