Store robots.txtRobots & Sitemap

robots.txt rules for an online store: filters, search and cart pages

Delivery from 1 dayRevisions includedQuality check
Packages
Payment protectionDetails
  • Revisions within the package are free.
  • If you do not respond within 3 days of delivery, the work is accepted automatically.
Order · $20

The money is reserved when you pay and released only once you accept the work.

robots.txt rules for an online store: filters, search and cart pages

Example result

Example for the Standard package

SEO audit: website of a regional plumbing company

Summary

AreaScoreMain issue
Indexing and crawlingNeeds work38 blog tags pages indexed, thin content
Technical healthFairMobile load time 4.8 s on service pages
On-pageNeeds work11 of 18 service pages share near-identical titles
Local signalsWeakTown pages are copies with the town name swapped
ContentFairBlog traffic is high, but posts do not link to services

Findings (excerpt)

1. Duplicate and weak titles (high priority)

11 service pages use the pattern "Plumbing Services | Northgate". Searchers and search engines cannot tell "boiler repair" from "drain unblocking".

PageCurrent titleSuggested title
/boiler-repairPlumbing Services, NorthgateBoiler Repair in [City], Same-Day Callout, Northgate
/blocked-drainsPlumbing Services, NorthgateBlocked Drains Cleared in [City] from £89, Northgate

2. Town pages are near duplicates (high priority)

The 7 town pages are 94% identical text. Each needs at least 250 words unique to that town: areas covered, typical jobs there, travel time, a local customer review.

3. Slow service pages on mobile (medium)

Hero images average 1.9 MB. Compressing to WebP and setting width to 1,200 px would bring load time to about 2.5 s.

4. Blog does not pass value to services (medium)

The top 10 posts get 62% of organic visits, but only 2 of them link to a service page. "Why is my boiler losing pressure?" alone gets about 1,100 visits a month and links nowhere.

5. Tag pages indexed (low)

38 /tag/ pages with 1–3 posts each. Set them to noindex.

Prioritised action plan

#ActionEffortImpactWho
1Rewrite titles and meta descriptions for 18 service pages1 dayHighContent
2Write unique content for 7 town pages3–4 daysHighContent
3Add service links to the top 10 blog posts2 hoursMediumContent
4Compress hero images2 hoursMediumDeveloper
5Noindex tag pages, update sitemap1 hourLowDeveloper

Re-check: crawl again 6 weeks after items 1–3 are live and compare impressions for "[service] + [town]" queries.

Client: Northgate Plumbing & Heating, 11 engineers, serving one city and 6 nearby towns Site: 214 pages: home, 18 service pages, 7 town pages, 160 blog posts, contact and legal pages Goal: more calls from local searches such as "boiler repair [town]"

The client and all details are made up.

About this service

Online stores generate endless crawlable URLs: filter combinations, sort orders, internal search results, cart, checkout, account and compare pages, and each platform names them differently. Blocking too little wastes crawl on junk pages, blocking too much hides products or the CSS that renders them. We read your current robots.txt and the URL patterns on your store and write a new file with commented rules for each pattern, plus notes on what should be handled with noindex or canonical instead.

What is included

  • What you getA ready robots.txt for your store with a commented rule for each URL pattern, a list of example URLs that the rules block and allow, and notes on patterns that need noindex or canonical rather than a block.
  • URL patternsFilters, sorting, search, cart and account
  • Commented fileEvery rule explained in the file
  • Keep openProducts, CSS, JS and chosen filter pages
  • Block or noindexWhen a block is the wrong tool
  • Test examplesURLs to check in a robots.txt tester

How it works

  1. 1
    PatternsYou send the store and URL examples
  2. 2
    RulesWe write and comment the file
  3. 3
    CheckYou get the file and test URLs

What we need from you

A link to the store, your platform, examples of filter, sort and search URLs from your store and the filter pages you want to keep indexable, such as brand or colour pages.

We will ask you:

  • Which platform runs the store?
  • Which filter pages should stay in search?
  • Paste a few filter, sort and search URLs from your store

Compare packages

PackagesBasicCore rules$10StandardFull store rules$20PremiumMulti-store rules$39
Deadline1 day2 days4 days
Revisions122
URL patterns coveredup to 8up to 20up to 40
Commented robots.txtYesYesYes
Test URL listYesYesYes
Order BasicOrder StandardOrder Premium

Payment protection

You pay when you order, but the money is released only for accepted work.

  • Fixed priceThe package price does not change after you pay.
  • The money is reservedWhen you pay, the amount is only reserved. It is released to the service once you accept the work.
  • Checked before deliveryEvery result goes through a quality check before it reaches you, and you see it on the order page before accepting it.
  • Free revisionsAsk for changes within your brief as many times as the package allows.
  • Acceptance after 3 daysIf within 3 days of delivery you neither accept the work nor ask for changes, it is accepted automatically. If you write in the order chat after delivery, the order waits; once we answer, we may start the 3 days again and will say so in the chat.
  • Redo or a full refundIf the result does not match your brief and the revisions did not fix it, or the work takes more than twice its deadline, write in the order chat: we will redo the work or refund the payment in full.
How it works →

Questions and answers

Can I edit robots.txt on Shopify?

Yes, through the robots.txt.liquid template. We write the rules in that form if your store is on Shopify.

Does blocking a page remove it from Google?

No. A blocked URL can still appear in results if other pages link to it. Pages that must disappear need noindex and must stay crawlable until they drop out.

Will you upload the file?

No, we do not log in to your store or server. You or your developer replace the file and test it.

Packages
Order · $20· 2 days