A Python parser that lists a podcast's episodes from its open site: number, title, date, length and link

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Podcast hosts moving to a new site, show-notes editors and researchers use this parser to list episodes in a table: number, season, title, date, length, short description, topics or tags, link to the episode page and the audio link when the page shows it. It reads the show's own open site, or its open RSS feed, which most podcasts have. Send the episode list page, one episode page and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. It saves links, not audio files, and has no column for guests' names. It reads at a polite pace, never logs in or gets around captchas or protection, and does not read social networks or listening apps that need an account. You decide whether the site's terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Episodes in orderNumber, season and date in separate columns
• Length in minutesWhen the page or feed shows it
• Episode and audio linksLinks only, no audio files
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   ShowYou give the episode list or feed and one saved episode page
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the latest episodes against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The podcast's site and a link to its episode list, or its open RSS feed address, one episode page and roughly how many episodes there are. The fields in the order of your table, e.g. number, season, title, date, length, description, tags, episode link, audio link. One or two of those pages saved from your browser (File > Save page as) or screenshots. CSV, Excel or JSON.

We will ask you:

• Which podcast?
• The fields you need
• How many episodes roughly?
• The results file

Compare packages

Questions and answers

Did not find your answer? Ask us