ogbuilds.ai/robot.guardsmart robots.txt that pays for itself
robot·guard turns the most overlooked file on your site into a control panel. tick the crawlers you want (search engines, archives), switch off ai scrapers, and robot·guard writes valid directives for each choice. preview the file, download it, drop it at your site root.
robot·guardUnderneath the toggles is a real editor: curated allow-lists, a maintained blocklist of AI user-agents like GPTBot and CCBot, plus your own rules. Preview the exact file as you go and download it ready to drop at your site root.
robot·guardbefore & after
ai scrapers eating your bandwidth
beforegptbot and ccbot crawl your site freely by default
afterswitch them off from a maintained blocklist in a click
writing robots.txt by hand
beforehunt for the right directives and hope the syntax is valid
aftertick the crawlers you want, valid directives written for you
keeping the file current
beforere-edit every time a new scraper appears
afterthe ai-scraper blocklist stays current, synced across your sites (pro)
simple pricing
Free
build a clean file in the editor
- whitelist good bots, block ai scrapers
- preview + download your robots.txt
Pro
your robots.txt, managed for good
everything in free, plus:
- hosted robots.txt url: always live, updates itself
- auto-block new ai scrapers as we add them
- email alerts when new crawlers appear
- unlimited configs synced across devices
a look inside
a few of the screens you'll actually use.
- whitelist the good bots
- block ai scrapers
- generate & download
robot·guardrobot·guardcurated ai scraper blocklist
kept current as new crawlers appear. toggle the ones you want shut out.
robot·guard guides
Ways to use robot·guard, and how it compares.
- use caserobot·guard gives you one place to whitelist the bots you want and block the ones you don'trobot·guard turns robots.txt into a control panel: whitelist Googlebot and crawlers that send traffic, block AI scrapers like GPTBot, preview and download.
- use caseWhat is robots.txt, and why it matters more than everWhat is robots.txt? A text file at your site root telling crawlers what they may visit. What it can't do, and why AI scrapers make robot·guard useful.
- how toHow to block specific AI bots from scraping your websiteBlock AI bots like GPTBot, ClaudeBot, CCBot and Google-Extended in robots.txt by user-agent, and let robot·guard keep the blocklist current for you.
- use caseThe hidden cost of unwanted bot traffic, and how AI scrapers inflate itCost of bot traffic: bots are roughly half of traffic, and AI scrapers give nothing back. How that hits bandwidth and hosting, and how robot·guard cuts it.
- how torobots.txt for SEO: how to whitelist Googlebot without locking out the restrobots.txt for SEO: keep Googlebot, Bingbot and key crawlers reaching what they should while blocking costly bots. robot·guard scopes every rule right.
- comparisonrobots.txt vs firewall: choosing the right bot protectionrobots.txt vs firewall: one asks compliant bots to stay out; one blocks any request. Most sites need both, and robot·guard handles the robots.txt half.