A Python parser that lists every article of a blog for a content audit: title, date, words and headings

Example result

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

About this service

Before you rewrite, merge or delete old posts, you need a list of what the blog has. This parser goes through a blog's open article list, opens each post and saves a row: link, title, date published and updated when shown, category, tags, word count, number of subheadings, pictures and links, and the meta description. It works best on your own blog and can read a competitor's open blog too. Send the blog address, one article and one or two pages saved from your browser. You get Python code, a settings file and a README and run it on your computer. It gives facts, not scores, and promises no search positions. It reads open pages at a polite pace, never logs in or gets around captchas or protection, and leaves author names out. You decide whether a site's terms allow it; a later layout change needs an update.

What is included

• What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
• Counts per articleWords, subheadings, pictures and links
• Dates and categoriesPublished, updated, category and tags
• Title and descriptionAs written in the page, with their length
• Ready for filtersSort by date or length in Excel
• Polite paceA pause between pages, set in the settings

How it works

1. 1
   BlogYou give the blog address and one saved article
2. 2
   ParserWe write the code and the guide to running it
3. 3
   RunYou run it and check the counts of two or three posts
4. 4
   ChangesYou ask for changes within your package and send any error message

What we need from you

The blog address and a link to its article list, one article and roughly how many posts there are. The fields you need, e.g. link, title, date, category, tags, word count, subheadings, meta description. One or two of those pages saved from your browser (File > Save page as). Whether it is your blog or another one, and CSV or Excel.

We will ask you:

• Which blog?
• The fields you need
• How many articles roughly?
• Anything else it should do

Compare packages

Questions and answers

Did not find your answer? Ask us