use case

robot.guard: one place to whitelist the bots you want and block the ones you don't

the short answer

robot.guard is a robots.txt manager: you tick the legitimate crawlers you want to keep, switch off the AI scrapers draining your bandwidth from a curated blocklist, add your own allow/disallow rules, preview the exact standards-compliant file, and download it to drop at your site root.

Your robots.txt is the first file almost every crawler on the internet reads, and for most sites it is a few lines someone pasted in years ago and never looked at again. That is a problem now that a new class of bots — AI scrapers harvesting the whole web for training data — hits sites harder than search engines ever did, while sending nothing back. robot.guard exists to give you deliberate control over that file without hand-editing syntax you have to get exactly right.

This page is the full tour: what robot.guard does, how the editor is built, and why fine-grained control over robots.txt is the cheapest performance and cost win most sites are leaving on the table.

~50%of all web traffic is automated bots, not people (Imperva, 2024)
start guarding your site →about robot.guard
robotguard.ogbuilds.ai
robot.guard
editorconfigsblocklist
download
whitelist good bots4 allowed
GooglebotGooglebot
BingbotBingbot
DuckDuckBotDuckDuckBot
Internet Archiveia_archiver
block ai scrapers3 blocked
GPTBot · OpenAIGPTBot
ClaudeBot · AnthropicClaudeBot
CCBot · Common CrawlCCBot
Bytespider · ByteDanceBytespider
robots.txt · live valid
# generated by robot.guard
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# blocked ai scrapers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
14 lines · 4 agentsupdates as you toggle

where this happens in the app

the editor is the whole product in one screen — tick the crawlers you want, switch off the ai scrapers you don't, and watch a valid robots.txt write itself on the right.

  1. 1whitelist the good bots — googlebot, bingbot, the internet archive — each toggle writes a valid allow block.
  2. 2block the ai scrapers from a curated list: gptbot, claudebot, ccbot, and the rest.
  3. 3the live preview is the exact file you download — it updates as you toggle, valid by construction.

The editor: toggles that write valid directives

The heart of robot.guard is the editor. On one side you have two curated lists — the good bots and the AI scrapers — each a row you flip on or off. Tick Googlebot, Bingbot, DuckDuckBot, or the Internet Archive to explicitly allow them; switch off GPTBot, ClaudeBot, CCBot, or Google-Extended to disallow them. Every toggle writes a correct, standards-compliant User-agent / Allow / Disallow block for you, so a missed colon or a wildcard in the wrong place never silently breaks your file.

On the other side is a live preview of the exact robots.txt the toggles produce, updating as you click. What you see is what you download — there is no separate export step that might drift from the screen. When it looks right, you grab the file and drop it at your site root.

A maintained blocklist, not a static template

New AI crawlers appear constantly, and a robots.txt you wrote last year does not know their user-agents. robot.guard keeps a curated blocklist of known AI scrapers — who operates each one and what it is for — so you can shut out model-training and answer-engine bots without hunting through documentation. You stay in control: nothing is blocked unless you choose to block it.

For everything the curated lists do not cover, custom rules let you add your own user-agents and path patterns — disallow a heavy /search endpoint, allow a partner's crawler, or carve out a directory. It is a real editor underneath the friendly toggles.

Why this is worth doing

Unwanted crawlers cost real money: bandwidth, origin CPU, cache pollution, and skewed analytics. Trimming the bots that take the most and give back nothing reduces server load and hosting bills while leaving the crawlers that actually send you traffic untouched. For a content site, that is the difference between paying to train someone else's model and keeping your resources for your readers.

how it works

  1. 01

    whitelist good bots

    Tick the crawlers you want to keep from a curated list of search, social, and archive bots.

  2. 02

    block ai scrapers

    Switch off known AI user-agents, or add your own rules for anything custom.

  3. 03

    preview & download

    Watch the exact robots.txt build live, then download it for your site root.

frequently asked

Do I need to know robots.txt syntax to use robot.guard?
No. You work in toggles and simple rules; robot.guard writes the valid directives. The live preview shows the exact file so you can learn the syntax if you want to.
Will blocking AI scrapers hurt my SEO?
No. AI training crawlers like GPTBot and CCBot are separate from search crawlers like Googlebot. robot.guard lets you allow search and social bots while blocking the AI ones, so your search visibility is untouched.
Does robot.guard host the file for me?
Not in this version. You download the generated robots.txt and place it at your site root. That keeps you in full control of where it lives and how it is served.
Can I manage more than one site?
Yes. Each site gets its own saved config, so you can keep separate robots.txt files for your main site, docs, and shop and edit them independently.

Last updated June 9, 2026

more on robot.guard

ready to try robot.guard?

start guarding your site →