What we check
After the page audit we fetch /robots.txt of the host and evaluate it as RFC 9309 describes: a crawler follows its own User-agent group when one exists, otherwise the * group; the longest matching Allow or Disallow wins, and Allow wins a tie. The decision is made for the exact audited path, not for the site in general.
The list: Googlebot and Bingbot; the AI search crawlers OAI-SearchBot, Claude-SearchBot and PerplexityBot; the agents that fetch when a person asks — ChatGPT-User, Claude-User and Perplexity-User; and the training crawlers GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot and meta-externalagent. A Content-Signal line (search, ai-input, ai-train), when declared, is shown too.
Then we request the page with the published user agent of Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User, GPTBot and ClaudeBot, and once as a browser. A 403, a 429 or a firewall challenge that only a crawler gets is a block that robots.txt does not show. We also read the newest lastmod of your sitemap: a sitemap that has not changed in 30 days tells crawlers there is nothing new. When Cloudflare is in front of the site, we show where to let the AI search crawlers in.
Three different choices
Blocking a training crawler does not remove the page from search or from AI answers, and blocking one company's search crawler does not stop another company's training. Sites that want visibility usually keep Googlebot, Bingbot and the AI search crawlers allowed and decide training on its own.
robots.txt is a public statement, not a lock. A firewall rule that blocks AI user agents does not show in the file, which is why we also request the page as each crawler. Two of the user-request agents say in their own documentation that they may ignore robots.txt. If robots.txt answers 5xx, Google treats the whole site as disallowed until it recovers, and the checker flags that.
What the audit of https://example.com/ returned on 23 September 2026
Before
GET https://example.com/robots.txt → 404
<!-- no file: every crawler on the list may fetch the page, training crawlers included -->After
User-agent: *
Content-Signal: search=yes, ai-train=no
Allow: /
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
<!-- search and AI search stay open; three training crawlers are kept out -->example.com has no robots.txt, and a missing file means everything is allowed. The second version is one common policy — stay in search and in AI answers, stay out of training. It is a choice, not a correction.
Questions
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot is OpenAI's training crawler. ChatGPT search results come from OAI-SearchBot, and pages opened on a user's request come through ChatGPT-User. Blocking GPTBot keeps your pages out of training; blocking OAI-SearchBot keeps them out of ChatGPT search.
What is Google-Extended?
Not a separate crawler. It is the robots.txt name Google checks before using content fetched by Googlebot for Gemini training and grounding. Blocking it does not affect Google Search.
Do AI crawlers obey robots.txt?
The companies behind the search and training crawlers listed here publish robots.txt names for them. ChatGPT-User and Perplexity-User fetch on a person's request and, by their own documentation, may ignore robots.txt. To enforce a block, use your firewall.
Can it see a firewall block?
Yes, by name. We request the page with each crawler's published user agent and show who gets a 403, a 429 or a challenge. Cloudflare recognises the real crawlers by their IP address, so a rule that trusts verified bots can treat the real crawler differently from our request.
How do I let the AI search crawlers in on Cloudflare?
In the Cloudflare dashboard, open your domain, then AI Crawl Control and its Crawlers list. Set OAI-SearchBot, Claude-SearchBot and PerplexityBot to Allow; training crawlers can stay on Block. Also check that Bot Fight Mode or a custom WAF rule does not challenge them.