Sitemap parserReady-to-Use

A ready-made Python parser that reads a site's sitemap.xml and saves each page's title, description and H1

Usually ready in 15 minutesRevisions includedQuality check
Packages
Payment protectionDetails
  • Revisions within the package are free.
  • If you do not respond within 3 days of delivery, the work is accepted automatically.

The money is reserved when you pay and released only once you accept the work.

A ready-made Python parser that reads a site's sitemap.xml and saves each page's title, description and H1

Example result

Example for the Standard package

Parser of a garden tools supplier's catalog into Excel

What the result looks like

ArticleNamePriceIn stockWeightPhoto
GT-1042Pruning shears, 20 mm18.90Yes0.24 kg…/gt-1042.jpg
GT-1187Folding saw, 210 mm24.50Yes0.31 kg…/gt-1187.jpg
GT-2210Hedge shears, wavy blade39.00No1.10 kg…/gt-2210.jpg
GT-3305Hand trowel, stainless9.80Yes0.18 kg…/gt-3305.jpg
GT-4120Garden hose 25 m, 1/2"42.00Yes2.90 kg…/gt-4120.jpg

The file in the example has 1,236 products from 52 list pages.

A part of the code

def product_links(list_url: str) -> Iterator[str]:
    """Product page links from every page of the list, following 'Next'."""
    url: str | None = list_url
    while url:
        page = fetch(url)
        for card in page.select("div.product-card a.title"):
            yield urljoin(url, card["href"])
        next_link = page.select_one("a.pagination-next")
        url = urljoin(url, next_link["href"]) if next_link else None


def fetch(url: str) -> BeautifulSoup:
    """One open page, at a polite pace: a pause between requests and a few tries on errors."""
    for attempt in range(3):
        time.sleep(settings.delay_seconds)
        response = session.get(url, timeout=30)
        if response.status_code == 200:
            return BeautifulSoup(response.text, "html.parser")
        log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
    raise ScraperError(f"{url} did not open after 3 tries")

Files: a ZIP archive with scraper.py, settings.ini (the start page, the pause between requests, the file name), requirements.txt and a README on installing and running it. It reads only open pages, one every 2 seconds, and checks the site's robots.txt first. The client runs it on their own computer.

Client: an online garden shop that resells a supplier's tools; the supplier has an open catalog on its website but no price list file. Task: go through the catalog's list pages, open every product page and save the article number, name, price, stock, weight and main photo link to Excel, to compare prices once a week.

The client and all details are made up.

About this service

A ready-made sitemap parser: a typical Python parser already worked out for sitemap.xml files, set up for the site you name, your own or any open one. It reads the list of addresses from the sitemap, opens each page and saves what you need for an inventory: the address, the date in the sitemap, the response status, title, meta description, H1 and, from Standard, canonical, the number of H2 headings, word count and pictures without alt text. The table shows at once which pages have no title, a doubled description or a missing H1, which helps before a redesign, a move or an SEO review. You send the site and the fields you need. You run it on your own computer. It sees only pages in the sitemap and open to visitors, does not judge SEO or give advice and promises no positions in search.

What is included

  • What you getA ZIP archive with the ready sitemap parser in Python, a settings file with the addresses and the pause between pages, and a README; the code is also shown on the order page.
  • From the sitemapEvery address listed in sitemap.xml
  • Page fields readyStatus, title, description, H1 and more
  • Gaps easy to seeEmpty or repeated titles stand out in the table
  • Polite paceA pause between pages, set in the settings
  • A plain guideInstalling Python and running it, step by step

How it works

  1. 1
    PagesYou give the site, its sitemap and the fields you need
  2. 2
    SetupWe fit the ready parser to those pages and write the guide
  3. 3
    RunYou run it and check the first rows of the table
  4. 4
    ChangesYou ask for changes within your package and send any error message

What we need from you

The site and the address of its sitemap.xml, usually at /sitemap.xml, and roughly how many pages it has. The fields you need in the order of your table, for example address, status, title, description, H1 and, from Standard, canonical and word count. The results file: CSV, Excel or, in Premium, JSON. For Premium, the other sites to compare and how often to run it.

We will ask you:

  • Which site?
  • The fields you need
  • How many pages roughly?
  • Results file and other sites

Compare packages

PackagesBasic$16Standard$33Premium$58
What is includedTitles of listed pagesSitemap index and pagesUp to 3 sites
Deadline1 day2 days3 days
Revisions123
Sites11up to 3
Kinds of page12 (list and item)up to 3
Fieldsup to 10up to 25up to 25 on each
Goes through the pages of a listNoYesYes
Results fileCSVExcel or CSVExcel, CSV or JSON
Duplicates removed, resumes where it stoppedNoNoYes
Ready-made versionSitemap and page titlesSitemap and page titlesSitemap and page titles
Order BasicOrder StandardOrder Premium

Payment protection

You pay when you order, but the money is released only for accepted work.

  • Fixed priceThe package price does not change after you pay.
  • The money is reservedWhen you pay, the amount is only reserved. It is released to the service once you accept the work.
  • Checked before deliveryEvery result goes through a quality check before it reaches you, and you see it on the order page before accepting it.
  • Free revisionsAsk for changes within your brief as many times as the package allows.
  • Acceptance after 3 daysIf within 3 days of delivery you neither accept the work nor ask for changes, it is accepted automatically. If you write in the order chat after delivery, the order waits; once we answer, we may start the 3 days again and will say so in the chat.
  • Redo or a full refundIf the result does not match your brief and the revisions did not fix it, or the work takes more than twice its deadline, write in the order chat: we will redo the work or refund the payment in full.
How it works →

Questions and answers

What if the site has no sitemap?

Then give a list of addresses in a text file instead; the parser reads them the same way. It does not crawl the whole site link by link.

Will it tell me how to fix my SEO?

No. It gives you the data in a table; what to change is your call or a task for an SEO audit. Nobody can promise positions in search.

How long does a big site take?

It reads pages at a polite pace with a pause you set, so a site of a few thousand pages takes an hour or more. In Premium it resumes where it stopped.

Am I allowed to collect this data?

Collecting public data is allowed in many cases, but sites' terms and data laws differ between countries, so you decide whether you may collect and use it. The parser never logs in, gets around protection or collects personal data.

Why does it cost less than a custom parser?

It is a ready-made, typical sitemap parser: its fields and the way it moves through pages are already worked out, and we fit it to the site you name and your columns. Unusual sites and fields are the custom parser service.

Did not find your answer?

Packages