Skip to content
GEO / AI visibility

AI Crawlers & Bots Directory

Twelve crawlers decide how AI systems see your website — from OpenAI's GPTBot and Anthropic's ClaudeBot to PerplexityBot and Bingbot. Each page explains what the bot does, what happens if you block it, and the exact robots.txt rule that controls it.

Why AI crawler access matters in 2026

ChatGPT search, Claude, Perplexity and Bing Copilot all answer with sources — and every source had to be fetched first by that engine's own crawler. When robots.txt or a firewall rule keeps a crawler out, the page never enters that engine's index, and a page an engine cannot retrieve is a page it cannot cite.

This runs separately from Google: Googlebot rules decide classic search indexing, while OAI-SearchBot, Claude-SearchBot, PerplexityBot and Bingbot decide whether the same content surfaces in AI answers. Each engine documents its own user-agent token, so robots.txt is the single file where you control who reads what — the directory below lists the exact token for every crawler.

How robots.txt controls AI crawlers

robots.txt sits at your domain root and is read before crawling starts. Each User-agent block names a crawler; the Allow and Disallow lines under it decide which paths that crawler may fetch, and a rule applies only to the agent it is written for — so you can allow one crawler while blocking another on the same site.

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /
Disallow: /private/

Test every edit with the free robots.txt checker, which re-reads your live file and flags conflicting or over-broad rules. If a wildcard Disallow: / is already keeping AI crawlers out, the unblock guide shows how to narrow it without exposing private paths.

GPTBotOpenAI
AI training

OpenAI's crawler that collects publicly available web pages to train its generative AI models. It is separate from ChatGPT-User (real-time fetches) and OAI-SearchBot (search indexing).

Read the guide →
OAI-SearchBotOpenAI
Search & answers

OpenAI's search crawler. It indexes the pages that power ChatGPT's search feature and the source links ChatGPT shows next to answers.

Read the guide →
ChatGPT-UserOpenAI
Real-time user fetch

Fetches pages on behalf of an individual ChatGPT user at their request — for example when ChatGPT opens a link you pasted into the conversation. It is not a bulk crawler.

Read the guide →
ClaudeBotAnthropic
AI training

Anthropic's crawler that gathers public web data for training and evaluating its Claude models. Anthropic identifies it through its user-agent token and controls it via robots.txt.

Read the guide →
Claude-SearchBotAnthropic
Search & answers

Anthropic's crawler for improving and grounding the sources Claude surfaces when it answers questions with web results.

Read the guide →
Claude-UserAnthropic
Real-time user fetch

Fetches pages in real time when an individual Claude user asks it to open or read a specific URL during a conversation.

Read the guide →
PerplexityBotPerplexity
Search & answers

Perplexity's crawler that maintains the search index behind its AI answer engine. Perplexity documents it as controllable through robots.txt.

Read the guide →
Google-ExtendedGoogle
AI controls token

Not a standalone search crawler — it is a controls token you add to robots.txt to manage whether Google may use your content for Gemini apps and Vertex AI grounding, independently of Google Search. Appearing in Google Search (and AI Overviews) follows Googlebot rules, not Google-Extended.

Read the guide →
Google-CloudVertexBotGoogle Cloud
Search & answers

Google Cloud's crawler that fetches content for Vertex AI Search services and customer-built search deployments on Google Cloud.

Read the guide →
BingbotMicrosoft
Search & answers

Microsoft's search crawler. Bing's index built by Bingbot also feeds Bing chat and Copilot answers, so it is the Bing equivalent of Googlebot.

Read the guide →
BingPreviewMicrosoft
Search & answers

Microsoft's rendering crawler that takes snapshots of pages so Bing can show link previews and rich snippets that match the rendered page.

Read the guide →
Applebot-ExtendedApple
AI controls token

A controls token for Apple's robots.txt: it governs whether your content may be used to train Apple's foundation models behind Apple Intelligence and Siri. General Applebot (Apple's search crawler) is a separate user-agent.

Read the guide →

Check your own robots.txt

Not sure whether your site allows these crawlers? Run your robots.txt through the checker — it flags missing or conflicting AI bot rules.

Open the Robots.txt Checker

Frequently Asked Questions

What happens if I block GPTBot?▾
Your content stops being used for OpenAI model training — ChatGPT search answers and user-requested browsing keep working, because those run on OAI-SearchBot and ChatGPT-User, which robots.txt treats as separate crawlers with their own rules. GPTBot details
What's the difference between GPTBot and OAI-SearchBot?▾
GPTBot is the bulk crawler that collects publicly available pages for training, while OAI-SearchBot is the search crawler that indexes the pages powering ChatGPT's search feature and the source links shown next to answers. Block GPTBot and training stops while search keeps working; block OAI-SearchBot and your pages stop appearing as sources and citations inside ChatGPT search answers. OAI-SearchBot details
Does robots.txt blocking remove my content from ChatGPT answers?▾
Blocking OAI-SearchBot does — your pages stop appearing as sources and citations inside ChatGPT search answers. Blocking GPTBot alone does not affect answers, and a page a user explicitly asks ChatGPT to open is fetched by ChatGPT-User, which is controlled by its own robots.txt rule.
How do I safely allow AI crawlers?▾
Allow each crawler by name, test the file with the free robots.txt checker, then verify the live file and monitor after every edit — allow rules take effect on the crawler's next scheduled visit, not instantly. Start from the example above, keep private paths disallowed, and re-check the file whenever you change it. Robots.txt checker · Unblock guide