Logo von nextlevels
Request a project
Check crawlers for free

Can GPTBot and ClaudeBot get in? The AI Crawler Check

The check reads your robots.txt and fetches your homepage live as GPTBot, ClaudeBot and PerplexityBot. You see whether a rule or a firewall locks them out.

  • Four requests: robots.txt and the homepage three times
  • No email address needed
  • Free, no sign-up

The live request uses the crawler's name as user agent but comes from our server. It does not detect blocks by IP address.

  • GPTBot
  • ClaudeBot
  • PerplexityBot
  • CCBot
  • Bytespider
  • RFC 9309

What the check covers

Two places decide whether an AI crawler gets your content: the rules in the robots.txt and what your server actually delivers to it.

  • robots.txt

    The check looks for a dedicated block for every AI crawler and reports Disallow: / as a block. Single blocked paths appear as a note.

  • Live request

    Three requests to your homepage with the user agents GPTBot, ClaudeBot and PerplexityBot. An error status or a suspiciously short response points to a block in the firewall, CDN or web server.

  • Global block

    User-agent: * with Disallow: / blocks every crawler without its own block, search engines included.

  • Training bots

    CCBot and Bytespider mainly collect training and raw data. Blocking them is a legitimate choice and is shown neutrally.

Limits: the live request comes from our server, not from the IP ranges of OpenAI, Anthropic or Perplexity. The check does not see blocks or allowances by IP address.

Sources: OpenAI: overview of OpenAI crawlers · Anthropic: ClaudeBot and robots.txt · Perplexity: Perplexity crawlers · RFC 9309 (robots.txt)

Every crawler at a glance

The rule in the robots.txt, your server's answer and the verdict – for every crawler.

Check crawlers
Result for example.comexample.com
Live request to the homepage
  • GPTBot: HTTP 403 – blocked
  • ClaudeBot: HTTP 200, 48312 bytes – served
  • PerplexityBot: HTTP 200, 48312 bytes – served
Rules in the robots.txt
  • GPTBotno block
  • ClaudeBotno block
  • PerplexityBotfully blocked
  • CCBotfully blocked

Blocking these bots is a legitimate choice and not an error.

Example with made-up values.

The full AI Visibility Check

Access is only the precondition. The AI Visibility Check also tests whether your content is citable: authority, schema, sitemap, llms.txt and technology – with a score and the three biggest levers.

Go to the AI Visibility Check

AI crawlers: frequent questions

Your question isn't here? Ask us directly

What are GPTBot, ClaudeBot and PerplexityBot?
They are the crawlers of OpenAI, Anthropic and Perplexity. They fetch web pages so that their content can be used in the providers' AI products. Each one identifies itself with its own user agent and can be controlled individually in the robots.txt.
How do I allow AI crawlers in the robots.txt?
Create a dedicated block for the crawler: a line “User-agent: GPTBot” with “Allow: /” below it. A crawler's own block takes precedence over the general rule under “User-agent: *”. The check gives you the matching lines to copy when it finds a block.
How do I block an AI crawler?
With a dedicated block and “Disallow: /”. The robots.txt is a request, not a technical barrier: it works for crawlers that respect it. Keep in mind that a blocked crawler cannot cite your content either.
The robots.txt allows everything, but the live request is still blocked. Why?
Then something in front of your website filters by user agent: a firewall, your CDN's bot protection or a rule in the web server. Check these places and exempt the AI crawlers you want to admit.
How accurate is the live request?
It shows how your server reacts to the crawler's user agent. But the request comes from our server and not from the providers' IP ranges. If your bot protection blocks or allows by IP address, the result for the real crawler can differ.

Find and remove the blocks

Send us your domain. We'll find where AI crawlers get stuck – robots.txt, CDN or web server – and tell you who should fix it: your team, your hosting provider or us.

Profile picture of Kevin Bolleßen, Head of BD & Digital Marketing
Kevin Bolleßen
Head of BD & Digital Marketing
info@next-levels.de +49 (0) 2161 539 71 60