AI crawler checker
Search, AI answers and AI training are three separate doors, and robots.txt opens or closes each one per crawler. This checker reads the file the way the crawlers do and tells you, for this URL, who gets in and which line decided it.
What we check
After the page audit we fetch /robots.txt of the host and evaluate it as RFC 9309 describes: a crawler follows its own User-agent group when one exists, otherwise the * group; the longest matching Allow or Disallow wins, and Allow wins a tie. The decision is made for the exact audited path, not for the site in general.
The list: Googlebot and Bingbot; the AI search crawlers OAI-SearchBot, Claude-SearchBot and PerplexityBot; the agents that fetch when a person asks — ChatGPT-User, Claude-User and Perplexity-User; and the training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. A Content-Signal line (search, ai-input, ai-train), when declared, is shown too.
Three different choices
Blocking a training crawler does not remove the page from search or from AI answers, and blocking one company's search crawler does not stop another company's training. Sites that want visibility usually keep Googlebot, Bingbot and the AI search crawlers allowed and decide training on its own.
robots.txt is a public statement, not a lock. A firewall rule that blocks AI user agents is invisible here, and two of the user-request agents say in their own documentation that they may ignore robots.txt. If robots.txt answers 5xx, Google treats the whole site as disallowed until it recovers, and the checker flags that.
What the audit of https://example.com/ returned on 23 September 2026
As found
GET https://example.com/robots.txt → 404
<!-- no file: every crawler on the list may fetch the page, training crawlers included -->Fixed
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
<!-- search and AI search stay open; three training crawlers are kept out -->example.com has no robots.txt, and a missing file means everything is allowed. The second version is one common policy — stay in search and in AI answers, stay out of training. It is a choice, not a correction.
Questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search results come from OAI-SearchBot, and pages opened on a user's request come through ChatGPT-User. Blocking GPTBot keeps your pages out of training; blocking OAI-SearchBot keeps them out of ChatGPT search.
What is Google-Extended?
Not a separate crawler. It is the robots.txt name Google checks before using content fetched by Googlebot for Gemini training and grounding. Blocking it does not affect Google Search.
Do AI crawlers obey robots.txt?
The companies behind the search and training crawlers listed here publish robots.txt names for them. ChatGPT-User and Perplexity-User fetch on a person's request and, by their own documentation, may ignore robots.txt. To enforce a block, use your firewall.
Why does my firewall block show as allowed?
Because this checker reads robots.txt. Blocks made at the edge (for example Cloudflare's AI bot blocking) happen before the file matters and are not visible from it.
See the full audit in your workspace