robot·guard gives you one place to whitelist the bots you want and block the ones you don't

The editor is the heart of it. On one side sit two curated lists, the good bots and the AI scrapers, each a row you flip on or off. Tick Googlebot, Bingbot, DuckDuckBot, or the Internet Archive to explicitly allow them; switch off GPTBot, ClaudeBot, CCBot, or Google-Extended to disallow them. Every toggle writes a correct, standards-compliant User-agent / Allow / Disallow block for you, so a missed colon or a wildcard in the wrong place never silently breaks your file.

On the other side is a live preview of the exact robots.txt the toggles produce, updating as you click. What you see is what you download. There's no separate export step that might drift from the screen. When it looks right, you grab the file and drop it at your site root.

New AI crawlers appear constantly, and a robots.txt you wrote last year doesn't know their user-agents. robot·guard keeps a curated blocklist of known AI scrapers, naming who operates each one and what it's for, so you can shut out model-training and answer-engine bots without hunting through documentation. Nothing is blocked unless you choose to block it.

For everything the curated lists miss, custom rules let you add your own user-agents and path patterns: disallow a heavy /search endpoint, allow a partner's crawler, carve out a directory. There's a real editor underneath the friendly toggles.

Unwanted crawlers cost real money: bandwidth, origin CPU, cache pollution, and skewed analytics. Trimming the bots that take the most and give back nothing reduces server load and hosting bills while leaving the crawlers that actually send you traffic untouched. For a content site, that is the difference between paying to train someone else's model and keeping your resources for your readers.