Cover both user-agents
Anthropic's web crawling shows up under more than one user-agent, and a complete opt-out names each one. Write one block as 'User-agent: ClaudeBot' followed by 'Disallow: /', and a second block as 'User-agent: anthropic-ai' followed by 'Disallow: /'. Each block is independent, so list both to be sure you are not leaving a gap. Save these in a plain text robots.txt file at your domain root.
If you only want to hold back part of your site, swap the single slash for a path, such as 'Disallow: /members/'. You can repeat Disallow lines under each user-agent to cover several sections. robot.guard keeps both ClaudeBot and anthropic-ai in its curated, maintained list, so toggling Anthropic adds the right names at once and you do not have to remember which agents Anthropic currently uses.
It is an opt-out Anthropic honours
Anthropic treats robots.txt as the mechanism for publishers to opt out of being crawled, and states that its crawlers obey it. So a correctly written block is respected by a compliant crawl. That said, robots.txt is a request rather than enforcement. It cannot physically block anything, and any crawler that chooses to ignore the standard will not be stopped by the file alone.
For that reason, treat robots.txt as your first and most important layer, then back it with a firewall or rate limiting for traffic that does not identify itself honestly or ignores the rules. The two work together: the file cleanly handles every well-behaved crawler, including ClaudeBot and anthropic-ai, while infrastructure-level controls catch the rest.
Your search ranking is not affected
Blocking Anthropic's crawlers does nothing to your search ranking. ClaudeBot and anthropic-ai are AI crawlers, completely separate from search crawlers like Googlebot, Bingbot, and DuckDuckBot. Google and Bing do not read or care about your Anthropic rules, so your indexing and ranking are untouched when you opt out.
robot.guard is built around exactly this separation. It whitelists the legitimate search and archive crawlers, Googlebot, Bingbot, DuckDuckBot, and the Internet Archive, while letting you block AI crawlers like ClaudeBot from a curated list. You preview the exact file live before shipping, so you can confirm at a glance that search is still allowed and only the AI agents are blocked.
how it works
- 01
Open your robots.txt
Open the robots.txt at your domain root, or create one as a plain text file so it resolves at example.com/robots.txt. In robot.guard you can start straight in the free editor.
- 02
Add the ClaudeBot block
Add 'User-agent: ClaudeBot' on one line and 'Disallow: /' on the next. To restrict only a section, use a path like '/members/' in place of the slash.
- 03
Add the anthropic-ai block
Add a second block: 'User-agent: anthropic-ai' then 'Disallow: /'. Both names matter. In robot.guard, toggling Anthropic inserts both user-agents at once from its maintained list.
- 04
Preview, download, and deploy
Preview the file to confirm Googlebot and Bingbot are still allowed, download robots.txt, and upload it to your site root. Verify by loading example.com/robots.txt in a browser.
frequently asked
- Why do I need to block both ClaudeBot and anthropic-ai?
- Anthropic's crawling appears under more than one user-agent. Blocking only ClaudeBot can leave anthropic-ai allowed. Listing both with their own Disallow rule gives a complete opt-out.
- Does Anthropic actually respect robots.txt?
- Anthropic documents robots.txt as the way to opt out and states its crawlers obey it. Since the file is a request and not enforcement, pair it with a firewall for any bots that ignore the standard.
- Will blocking Claude's crawlers hurt my SEO?
- No. ClaudeBot and anthropic-ai are separate from search crawlers like Googlebot and Bingbot. Blocking them has no effect on how your pages are indexed or ranked in search.
- Can I block Anthropic from only part of my site?
- Yes. Under each Anthropic user-agent, use a path such as 'Disallow: /blog/' instead of '/'. Add multiple Disallow lines to cover several sections while leaving the rest crawlable.
Last updated June 9, 2026