AI Crawlers & Bots Directory

OpenAI's crawler that collects publicly available web pages to train its generative AI models. It is separate from ChatGPT-User (real-time fetches) and OAI-SearchBot (search indexing).

OpenAI's search crawler. It indexes the pages that power ChatGPT's search feature and the source links ChatGPT shows next to answers.

Fetches pages on behalf of an individual ChatGPT user at their request — for example when ChatGPT opens a link you pasted into the conversation. It is not a bulk crawler.

Anthropic's crawler that gathers public web data for training and evaluating its Claude models. Anthropic identifies it through its user-agent token and controls it via robots.txt.

Anthropic's crawler for improving and grounding the sources Claude surfaces when it answers questions with web results.

Fetches pages in real time when an individual Claude user asks it to open or read a specific URL during a conversation.

Perplexity's crawler that maintains the search index behind its AI answer engine. Perplexity documents it as controllable through robots.txt.

Not a standalone search crawler — it is a controls token you add to robots.txt to manage whether Google may use your content for Gemini apps and Vertex AI grounding, independently of Google Search. Appearing in Google Search (and AI Overviews) follows Googlebot rules, not Google-Extended.

Google Cloud's crawler that fetches content for Vertex AI Search services and customer-built search deployments on Google Cloud.

Microsoft's search crawler. Bing's index built by Bingbot also feeds Bing chat and Copilot answers, so it is the Bing equivalent of Googlebot.

Microsoft's rendering crawler that takes snapshots of pages so Bing can show link previews and rich snippets that match the rendered page.

A controls token for Apple's robots.txt: it governs whether your content may be used to train Apple's foundation models behind Apple Intelligence and Siri. General Applebot (Apple's search crawler) is a separate user-agent.