Allow the bots that bring customers
The crawlers worth welcoming are the ones that put you in front of people: Googlebot and Bingbot for search, plus the social crawlers (the ones Facebook, LinkedIn and X use) that build the link preview when someone shares your page. Make sure none of these are accidentally disallowed, and add a Sitemap: line so they can find your pages, starting with the ones that sell: products, services, contact.
This is the half of robots.txt that supports growth. Blocking it by mistake is the one error that really hurts a small business, which is why a tool that won't let you fumble the syntax is worth more here than anywhere.
Block the bots that only cost you
Then turn away what takes without giving back. AI scrapers crawl small sites the same as large ones and add to your bandwidth bill for zero return, so disallowing the common AI user-agents is an easy saving. It's also good hygiene to disallow crawling of the pages nobody searches for: admin, login, cart, checkout. Not as real security (that's what authentication is for), but to keep those URLs out of search results where they don't belong.
robot·guard is built for exactly this owner: pick the good bots, switch off the scrapers from a maintained list, add a couple of custom disallows for admin paths, preview, and download. No syntax, no stale template, no security expertise. Just a correct file you can set and revisit a couple of times a year.
how it works
- 01
welcome search & social
Allow Googlebot and Bingbot plus the social preview crawlers, then add your sitemap.
- 02
block ai scrapers
Toggle off the common AI crawlers to stop paying to serve them.
- 03
hide admin paths
Disallow the URLs people only reach after clicking buy: cart, checkout, account login.
- 04
download & revisit
Place the file at your root and recheck a couple of times a year.
frequently asked
Is robots.txt a security feature?
Not really. It controls crawling, not access, so anyone can still visit a disallowed page directly. Use it to manage bots and tidy search results, and use authentication for real protection.
Do I need robots.txt if my site is small?
It's optional, but worth it. Even small sites get crawled by AI scrapers, and a five-minute file cuts that cost and keeps admin URLs out of search.
Could a robots.txt mistake hurt my business?
Yes. Accidentally disallowing your content can drop you from search. That's the main risk, and the one a generator that writes correct rules takes off the table.
Can I manage my shop and main site separately?
Yes. robot·guard saves a separate config per site, so your storefront and main site each get the right rules.
Last updated June 9, 2026