how to

How to check if AI crawlers can access your site (and why they might not)

the short answer

To check whether AI crawlers can access your site, read your robots.txt for rules targeting GPTBot, ClaudeBot, PerplexityBot, and Google-Extended — seo·check does this from a URL and reports each AI crawler as allowed or blocked, so you can catch an accidental block.

If you want AI answer engines to cite your pages, they first have to be allowed to fetch them. That permission lives in your robots.txt, and it's easy to get wrong: a broad disallow, a CDN's default rule, or a privacy-minded setting can block GPTBot or ClaudeBot without anyone noticing until traffic from AI answers never appears.

Checking it by hand means knowing every AI crawler's user-agent and reading robots.txt rule by rule. seo·check does that for you. Here's how to check whether AI crawlers can reach your site and what to do if one is blocked.

4 user-agentsAI crawlers seo·check looks up in your robots.txt
check a url →about seo·check

Why a crawler gets blocked by accident

robots.txt rules are matched per user-agent, and the AI crawlers each have their own: GPTBot, ClaudeBot, PerplexityBot, Google-Extended. A site can allow Googlebot perfectly while a separate, often forgotten, block applies to these — sometimes added deliberately during an AI-scraping scare, sometimes inherited from a host or template default.

The result is a page that ranks fine in classic search but is invisible to AI answers, with no error to alert you. The only way to know is to read robots.txt against each AI user-agent specifically, which is exactly the check most people skip.

What seo·check reports

seo·check fetches your robots.txt and evaluates it against each AI crawler's user-agent, then reports a clear allowed or blocked for each one. If a crawler is blocked, you can see whether it's a targeted rule or a broad disallow catching it, so you know exactly what to change.

It's a read-only check — seo·check reports the state of your robots.txt, it doesn't edit it. Once you've fixed the rule on your own site, re-run the check to confirm the crawler now shows as allowed.

how it works

  1. 01

    paste your URL

    Enter any page on your site — seo·check finds the robots.txt for that domain.

  2. 02

    run the check

    It evaluates robots.txt against each AI crawler's user-agent.

  3. 03

    read allowed / blocked

    See GPTBot, ClaudeBot, PerplexityBot, and Google-Extended each marked allowed or blocked.

  4. 04

    fix and re-check

    If one's blocked, update the rule on your site, then run the check again to confirm.

frequently asked

Which AI crawlers should I allow?
The common ones are GPTBot (ChatGPT), ClaudeBot (Claude), PerplexityBot (Perplexity), and Google-Extended (Google's AI surfaces). seo·check checks each so you can see which are allowed and decide which to open up.
Why would an AI crawler be blocked if I never blocked it?
Often a broad disallow, a host or CMS default, or a rule added during an AI-scraping scare catches the AI user-agents. Because there's no error, it goes unnoticed — which is why a dedicated check is useful.
Does allowing AI crawlers hurt my classic SEO?
No. AI crawler rules are separate from Googlebot's, so allowing GPTBot or ClaudeBot doesn't change how your site ranks in classic search — it only affects whether AI engines can read the page.
Will seo·check change my robots.txt for me?
No. It's a read-only checker — it reports whether each crawler is allowed or blocked. You make the change on your own site, then re-check.

Published May 18, 2026 · Last updated June 13, 2026

more on seo·check

seo·check — score your page for search + ai answer engines →

ready to try seo·check?

check a url →