A Python parser that collects open recipes into a table: title, category, time, servings, ingredients and link

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Food bloggers moving their own recipes, cooking apps and meal-planning services use this parser to build a recipe table from open pages: title, category, cuisine, cooking time, servings, ingredients, steps, calories when the page shows them, picture link and link. Many recipe pages also keep these facts in a hidden structured block, and the parser reads it when it is there. Send the site, a category and a recipe page and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. Recipe texts and photos belong to their authors: use only what you may. Author names are left out. It reads open pages at a polite pace and never logs in or gets around captchas or protection. You decide whether the site's terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Ingredients as a listOne ingredient per line in a cell, or a separate sheet
• Time and servingsAs numbers you can filter
• Picture linksLinks only; use photos you may
• Polite paceA pause between pages, set in the settings
• Runs on your computerWindows, macOS or Linux, with Python

How it works

1. 1
   PagesYou give the site, a category and one saved recipe
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check two recipes against the site
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The recipe site and the exact pages: a link to a category and to one recipe, and roughly how many recipes. The fields you need, e.g. title, category, cuisine, time, servings, ingredients, steps, calories, picture link. How ingredients should look: one cell or a separate sheet. One or two of those pages saved from your browser (File > Save page as). And whether these are your own recipes or another site's.

We will ask you:

• Which site and pages?
• The fields you need
• How many recipes roughly?
• How should ingredients look?

Compare packages

Questions and answers

Did not find your answer? Ask us