A Python parser for open property listings: price, area, rooms, district and link in a table, no contacts

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

For an agency, an investor or a buyer, this parser gathers open property listings into one table: type, price, price per square metre when shown, area, rooms, floor, city and district, street as the listing gives it, date posted, link and photo links. Set up the search on the site with your filters and send us that link, one listing and one or two pages saved from your browser. You get Python code, a settings file and a README, and you run it on your computer. People's data stays out: no names, emails or phone numbers of owners or agents, only the link to the listing. It reads open pages at a polite pace and never logs in or gets around captchas or protection. You decide whether the site's terms allow it, and a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Your search, your filtersStarts from a search link you set up on the site
• Numbers you can sortPrice, area and floor in separate columns
• No contactsNames and phones of sellers and agents are left out
• Photo linksLinks to the listing photos, not the files
• Polite paceA pause between pages, set in the settings

How it works

1. 1
   SearchYou set up the search and send its link and a saved listing
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the first rows against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The property site and a link to a search with your filters (city, type, price range), one listing page and roughly how many listings the search shows. The fields you need in the order of your table, e.g. type, price, area, rooms, floor, district, date posted, link. One or two of those pages saved from your browser (File > Save page as) or screenshots. CSV or Excel.

We will ask you:

• Which site and search?
• The fields you need
• How many listings roughly?
• Anything else it should do

Compare packages

Questions and answers

Did not find your answer? Ask us