Reference
AI crawlers, and how to let the right ones in.
The robots.txt names of the crawlers behind ChatGPT, Claude, Perplexity, Google, Bing and Apple, what each one is for, and copy-and-paste robots.txt rules to let AI answer engines in while keeping training crawlers out.
The AI crawlers.
Two kinds matter. Answer crawlers fetch pages to answer people's questions: block them and you can't be found or quoted in those answers. Training crawlers collect pages to train AI models: blocking them is your choice, and doesn't keep you out of AI answers.
| robots.txt name | Company | What it’s for | Kind |
|---|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search | Answers |
| ChatGPT-User | OpenAI | ChatGPT, when someone asks it to look | Answers |
| Claude-SearchBot | Anthropic | Claude search | Answers |
| Claude-User | Anthropic | Claude, when someone asks it to look | Answers |
| PerplexityBot | Perplexity | Perplexity | Answers |
| Googlebot | Google Search and AI Overviews | Answers | |
| Bingbot | Microsoft | Bing and Copilot | Answers |
| Applebot | Apple | Apple (Siri and Spotlight) | Answers |
| GPTBot | OpenAI | OpenAI training | Training |
| ClaudeBot | Anthropic | Anthropic training | Training |
| Google-Extended | Google Gemini training | Training | |
| Applebot-Extended | Apple | Apple training | Training |
| CCBot | Common Crawl | Common Crawl | Training |
| meta-externalagent | Meta | Meta training | Training |
Use the name in the first column in robots.txt, exactly as written.
Source: each company's own crawler documentation, checked 9 November 2026.
robots.txt rules you can copy.
Let AI answer engines in, keep training crawlers out. The most common choice for a business that wants to be found:
# AI training crawlers: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: CCBot
User-agent: meta-externalagent
Disallow: /
# Everyone else, including the AI answer crawlers: allowed
User-agent: *
Allow: /
Let everything in. If you're happy for AI models to train on your public pages too:
User-agent: *
Allow: /
A crawler follows the most specific group that names it, and only that group. So if you give a crawler its own group, repeat any rules from User-agent: * that you still want it to follow, such as keeping it out of your basket and account pages.
Where AI crawlers get stopped.
Firewalls and bot protection
Some CDNs and firewalls, Cloudflare among them, can block AI crawlers with one setting. Check which crawlers it covers, so you don't block the answer crawlers by accident.
Pages that need JavaScript
Most AI crawlers don't run JavaScript. If your content is drawn in the browser, they see an empty page.
noindex
A noindex tag or header keeps a page out of search engines, and out of the AI answers built on them.
Free tool
Check how visible your site is to AI.
Our free AI visibility checker reads your robots.txt and tells you which AI crawlers you're letting in, with a score out of 100.
What people ask us.
Does blocking GPTBot keep me out of ChatGPT?
No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches a page when someone asks it to. Block GPTBot and allow the other two, and you can still appear in ChatGPT's answers.
Does Google-Extended affect my Google rankings?
No. Google-Extended only controls whether Google may use your pages to train and ground its Gemini models. Google Search, including AI Overviews, uses Googlebot.
Do crawlers have to obey robots.txt?
robots.txt is a request, not a lock. The major AI companies say their crawlers follow it. To keep content private, put it behind a login.
How often does this list change?
New crawlers appear every few months. We check the list against each company's own documentation every month; the date it was last checked is under the table.