A Python parser for an open online course catalogue: title, school, price, length and format in a table

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Online schools, course aggregators and people who compare training use this parser to put an open course catalogue into a table: course title, school, topic, level, format, length, start date, price and old price, number of lessons, rating as shown and link. Send the catalogue page, one course page and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. It copies what the pages say; it does not judge course quality or promise results of training. Teachers' names and students' reviews are left out, so the table holds no personal data. It reads open pages at a polite pace and never logs in or gets around captchas or protection. You decide whether the site's terms allow it; a later change of its layout needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Prices to comparePrice, old price and instalment as separate columns
• Length and formatWeeks or hours, live or recorded, as the page says
• Topic and levelIn separate columns for filters
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   CatalogueYou give the catalogue and one saved course page
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check a few courses against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the exact pages: a link to the course catalogue and to one course page, and roughly how many courses there are. The fields in the order of your table, e.g. title, school, topic, level, format, length, start date, price, old price, link. One or two of those pages saved from your browser (File > Save page as) or screenshots. CSV or Excel.

We will ask you:

• Which site and pages?
• The fields you need
• How many courses roughly?
• Only some courses?

Compare packages

Questions and answers

Did not find your answer? Ask us