Payment protectionDetails
- Revisions within the package are free.
- If you do not respond within 3 days of delivery, the work is accepted automatically.
The money is reserved when you pay and released only once you accept the work.
Example result
Parser of a garden tools supplier's catalog into Excel
What the result looks like
| Article | Name | Price | In stock | Weight | Photo |
|---|---|---|---|---|---|
| GT-1042 | Pruning shears, 20 mm | 18.90 | Yes | 0.24 kg | …/gt-1042.jpg |
| GT-1187 | Folding saw, 210 mm | 24.50 | Yes | 0.31 kg | …/gt-1187.jpg |
| GT-2210 | Hedge shears, wavy blade | 39.00 | No | 1.10 kg | …/gt-2210.jpg |
| GT-3305 | Hand trowel, stainless | 9.80 | Yes | 0.18 kg | …/gt-3305.jpg |
| GT-4120 | Garden hose 25 m, 1/2" | 42.00 | Yes | 2.90 kg | …/gt-4120.jpg |
The file in the example has 1,236 products from 52 list pages.
A part of the code
def product_links(list_url: str) -> Iterator[str]:
"""Product page links from every page of the list, following 'Next'."""
url: str | None = list_url
while url:
page = fetch(url)
for card in page.select("div.product-card a.title"):
yield urljoin(url, card["href"])
next_link = page.select_one("a.pagination-next")
url = urljoin(url, next_link["href"]) if next_link else None
def fetch(url: str) -> BeautifulSoup:
"""One open page, at a polite pace: a pause between requests and a few tries on errors."""
for attempt in range(3):
time.sleep(settings.delay_seconds)
response = session.get(url, timeout=30)
if response.status_code == 200:
return BeautifulSoup(response.text, "html.parser")
log.warning("%s answered %s, try %d of 3", url, response.status_code, attempt + 1)
raise ScraperError(f"{url} did not open after 3 tries")
Files: a ZIP archive with scraper.py, settings.ini (the start page, the pause between requests, the file name), requirements.txt and a README on installing and running it. It reads only open pages, one every 2 seconds, and checks the site's robots.txt first. The client runs it on their own computer.
Client: an online garden shop that resells a supplier's tools; the supplier has an open catalog on its website but no price list file. Task: go through the catalog's list pages, open every product page and save the article number, name, price, stock, weight and main photo link to Excel, to compare prices once a week.
The client and all details are made up.
About this service
PR people, analysts and editors use this parser to keep a table of what open news sites publish: headline, date and time, section, tags, the short lead shown in the list and the link. Send the news list pages you follow, one news page and one or two pages saved from your browser. You get Python code with a settings file and a README, and you run it on your computer whenever you want a fresh list. It takes headlines and leads, not whole articles, since the texts belong to their publishers, and it leaves bylines out, so the table holds no names of people. It reads open pages at a polite pace and never logs in, gets around paywalls, captchas or protection. You decide whether the site's terms allow it; a later change of its layout needs an update.
What is included
- What you getA ZIP archive with the Python code, a settings file and a README with step-by-step instructions; the code is also shown on the order page.
- Headline and leadAs the news list shows them
- Date in one formatSo you can sort and filter by day
- Only your topicsAn optional filter by words you list
- Polite paceA pause between pages, set in the settings
- Runs on your computerWindows, macOS or Linux, with Python
How it works
- 1SourcesYou give the news lists and one saved news page
- 2ParserWe write the code and the guide to running it
- 3RunYou run it and check today's headlines against the site
- 4ChangesYou ask for changes within your package and send any error message
What we need from you
The news sites and the exact list pages or sections you follow, and one news page from each. The fields you need, e.g. headline, date, time, section, tags, lead, link. Words to keep or drop, if you want a filter. One or two of those pages saved from your browser (File > Save page as) or screenshots. How often you will run it, and CSV or Excel.
We will ask you:
- Which news sites and sections?
- The fields you need
- Words to keep
- How often will you run it?
Compare packages
| Packages | BasicHeadlines from one list$21 | StandardPages of the list and each item$42 | PremiumUp to 3 news sites$74 |
|---|---|---|---|
| Deadline | 1 day | 2 days | 3 days |
| Revisions | 1 | 2 | 3 |
| Sites | 1 | 1 | up to 3 |
| Kinds of page | 1 | 2 (news list and news page) | up to 3 |
| Fields | up to 10 | up to 25 | up to 25 on each |
| Goes through the pages of a list | No | Yes | Yes |
| Results file | CSV | Excel or CSV | Excel, CSV or JSON |
| Duplicates removed, resumes where it stopped | No | No | Yes |
| Order Basic | Order Standard | Order Premium |
Payment protection
You pay when you order, but the money is released only for accepted work.
- Fixed priceThe package price does not change after you pay.
- The money is reservedWhen you pay, the amount is only reserved. It is released to the service once you accept the work.
- Checked before deliveryEvery result goes through a quality check before it reaches you, and you see it on the order page before accepting it.
- Free revisionsAsk for changes within your brief as many times as the package allows.
- Acceptance after 3 daysIf within 3 days of delivery you neither accept the work nor ask for changes, it is accepted automatically. If you write in the order chat after delivery, the order waits; once we answer, we may start the 3 days again and will say so in the chat.
- Redo or a full refundIf the result does not match your brief and the revisions did not fix it, or the work takes more than twice its deadline, write in the order chat: we will redo the work or refund the payment in full.
Questions and answers
Can it collect the full articles?
No. Article texts belong to their publishers, so the parser takes the headline, the short lead from the list and the link; the full text you read on the site.
Can it read sites with a paywall?
Only what any visitor sees without signing in, often the headline and lead. It never logs in or gets around a paywall, captcha or protection.
Can it read Google News or news in social networks?
No. Google, Amazon, LinkedIn and social networks are out: their terms forbid it. Read the news sites themselves, as long as their terms allow it, which you decide.
How often can I run it?
A few times a day is fair for most sites at a polite pace. Premium adds a guide to running it on a schedule. After a change of a site's layout, the parser needs an update.
Did not find your answer?