一個 Python 爬蟲，把網站各頁面的 SEO 資料存成表格：title、description、H1、狀態、canonical

What the result looks like

Article	Name	Price	In stock	Weight	Photo
GT-1042	Pruning shears, 20 mm	18.90	Yes	0.24 kg	…/gt-1042.jpg
GT-1187	Folding saw, 210 mm	24.50	Yes	0.31 kg	…/gt-1187.jpg
GT-2210	Hedge shears, wavy blade	39.00	No	1.10 kg	…/gt-2210.jpg
GT-3305	Hand trowel, stainless	9.80	Yes	0.18 kg	…/gt-3305.jpg
GT-4120	Garden hose 25 m, 1/2"	42.00	Yes	2.90 kg	…/gt-4120.jpg

The file in the example has 1,236 products from 52 list pages.

A part of the code

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

Files: a ZIP archive with scraper.py, settings.ini (the start page, the pause between requests, the file name), requirements.txt and a README on installing and running it. It reads only open pages, one every 2 seconds, and checks the site's robots.txt first. The client runs it on their own computer.