AI crawler access decides whether bots like GPTBot, ClaudeBot and PerplexityBot are allowed to fetch your pages — and if they cannot fetch, your content cannot be cited in ChatGPT, Claude or Perplexity answers. The decision lives in robots.txt, but CDN and firewall rules can override it, so it has to be verified rather than assumed. Allow the AI search crawlers, make a separate deliberate choice about training crawlers, then confirm it with a scan.
Generative Engine Optimization (GEO) and Answer Engine Optimization (AEO) both end at the same bottleneck: an AI system has to fetch your page before it can quote it. ChatGPT search, Claude's web answers, Perplexity and Copilot all run on crawlers with their own names, their own robots.txt tokens and their own crawl schedules — none of them inherit Googlebot's access.
That makes access the cheapest win in GEO. Content that is blocked gets zero citations no matter how good it is, while content that is allowed still has to earn the citation through structure, schema and clarity. Fix access first: it is a one-file change with an immediate effect on whether you are eligible to appear at all.
Each crawler also does a different job. Search crawlers decide your citations, training crawlers decide whether your content feeds a model, real-time fetchers open links a user pasted into a chat, and controls tokens let you opt out of a vendor's AI features without touching search. The per-bot pages document all 12 in detail; this page is about the access decision itself and how to verify it.
OpenAI's crawler that collects publicly available web pages to train its generative AI models. It is separate from ChatGPT-User (real-time fetches) and OAI-SearchBot (search indexing).
How GPTBot is controlled →OAI-SearchBotOpenAIOpenAI's search crawler. It indexes the pages that power ChatGPT's search feature and the source links ChatGPT shows next to answers.
How OAI-SearchBot is controlled →ClaudeBotAnthropicAnthropic's crawler that gathers public web data for training and evaluating its Claude models. Anthropic identifies it through its user-agent token and controls it via robots.txt.
How ClaudeBot is controlled →PerplexityBotPerplexityPerplexity's crawler that maintains the search index behind its AI answer engine. Perplexity documents it as controllable through robots.txt.
How PerplexityBot is controlled →Google-ExtendedGoogleNot a standalone search crawler — it is a controls token you add to robots.txt to manage whether Google may use your content for Gemini apps and Vertex AI grounding, independently of Google Search. Appearing in Google Search (and AI Overviews) follows Googlebot rules, not Google-Extended.
How Google-Extended is controlled →BingbotMicrosoftMicrosoft's search crawler. Bing's index built by Bingbot also feeds Bing chat and Copilot answers, so it is the Bing equivalent of Googlebot.
How Bingbot is controlled →The remaining crawlers — ChatGPT-User, Claude-SearchBot, Claude-User, Google-CloudVertexBot, BingPreview and Applebot-Extended — are documented in the full AI bots directory.
Open /robots.txt and list the groups: one per matching user-agent token plus the wildcard. Look for an explicit rule for each AI crawler, then check whether the wildcard group disallows paths you want cited.
robots.txt controls fetching. A page can be fetched and still excluded by a <meta name="robots" content="noindex" tag or an X-Robots-Tag response header. Check both before concluding anything from robots.txt alone.
The free scan at /ai-visibility checks whether GPTBot, ClaudeBot and PerplexityBot are allowed by your robots.txt, whether an llms.txt file exists, and whether schema and entity signals are present — and scores those signals 0–100. Be clear about what it is not: it reads your public files and markup, it does not query ChatGPT, Perplexity or Claude, and it cannot report whether you currently appear in their answers.
The last word is a request that either succeeds or fails, and a log line that either exists or does not. Fetch the page with the crawler's user-agent, then search your server or CDN logs for those user-agents over the last few weeks.
curl -A "Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)" \
-o /dev/null -s -w "%{http_code}\n" https://example.com/pricingA 200 means the fetch was allowed at the edge and the origin. A 403, 429 or challenge page means something outside robots.txt is in the way.
robots.txt is a plain file at your domain root, split into groups. Each group starts with a User-agent: line and collects Allow and Disallow paths. A crawler uses the group whose token matches it — including multiple tokens in one group — and falls back to * when nothing matches. Longer, more specific path rules win over shorter ones.
The pattern most content sites want: explicit groups for the crawlers that build AI citations, and a wildcard group that keeps private paths closed for everyone.
# Allow the crawlers that build AI answers and citations User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Bingbot Allow: / # Your own wildcard group: public content open, private areas closed User-agent: * Allow: / Disallow: /account/ Disallow: /admin/ Disallow: /internal-search/ Sitemap: https://example.com/sitemap.xml
Blocking training crawlers only. OAI-SearchBot, Claude-SearchBot and PerplexityBot are untouched, so answers and citations keep working while your content stops being used for training.
# Opt out of model training only — answers and citations keep working User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / # Controls token: Gemini apps + Vertex AI, not Google Search User-agent: Google-Extended Disallow: /
Every vendor publishes its own crawlers, and new ones appear over time — the bot directory lists the twelve we track, each with its exact robots.txt rule.
# Too broad: this removes you from every AI answer surface User-agent: * Disallow: /
This one file would block every crawler without a dedicated group — AI search, training bots and real-time fetchers alike.
A single "User-agent: *" group with "Disallow: /" looks like a safe default, but it removes you from every AI answer surface — ChatGPT, Claude, Perplexity and Copilot included — and from Bing. It is also weak protection: scrapers that ignore robots.txt are unaffected, so you give up visibility without gaining security.
# Too broad: this removes you from every AI answer surface User-agent: * Disallow: /
Most sites only ever wrote rules for Googlebot. Under RFC 9309 a crawler uses the group whose user-agent token matches it, and falls back to "*" when none matches — so a broad Disallow you added for another reason silently applies to GPTBot, ClaudeBot and PerplexityBot too. Name each AI crawler explicitly if you have made a deliberate decision about it.
Allowing GPTBot while blocking OAI-SearchBot keeps training open and kills ChatGPT citations — usually the opposite of what a team intends. The same trap exists at Google: Google-Extended governs Gemini apps and Vertex AI, never Google Search or AI Overviews. Decide per crawler, not per company.
robots.txt is re-fetched on each crawler's own schedule and can also be cached at the CDN edge for hours. After a change, a fix can look ignored for a day or more. Verify the live file, purge the CDN cache, then wait for the next crawl before drawing conclusions.
robots.txt is an access convention, not authentication. It never protects private data — keep those routes behind login and server-side rules, and use rate limiting for anything a crawler could hammer. Public marketing pages want the opposite: open access for the crawlers that cite you.
robots.txt is only the first gate. Anything between the crawler and your origin can refuse the request on its own, and none of those layers read robots.txt the way a crawler does.
Cloudflare, Fastly and similar services score unfamiliar AI user-agents as suspicious and answer with a challenge or a 403. Your robots.txt says allow; the edge says no.
AI crawlers fetch in bursts. Tight per-IP or per-UA rate rules return 429, and a crawler that keeps getting 429 backs off — so the traffic you expected simply stops arriving.
WordPress and server security plugins ship presets that block AI crawlers by user-agent, and some also block the real-time fetchers, so links users paste into ChatGPT fail to open.
Origin firewalls, IP allowlists and geo restrictions apply before your application ever sees the request. A crawler reaching your origin from an unusual range can be dropped outright.
The fix is to test each layer: a direct request with the crawler's user-agent (shown above), then the same check from the crawler's own network where you can observe it, and finally your CDN's bot logs. Robots.txt, CDN rules and noindex tags have to agree before access is real.
A starting position for a public content site. The decision column is a recommendation, not a rule — training and controls tokens are policy choices, and each one has a per-crawler page explaining the exact trade-off.
| Crawler | What it does | Recommended | Example rule |
|---|---|---|---|
AI search indexing — the bots that decide whether you get cited Blocking them removes your pages from ChatGPT, Claude, Perplexity, Copilot and Vertex AI answers. Bingbot also carries your classic Bing rankings. | Allow | User-agent: PerplexityBot
Allow: / | |
Real-time fetches when a user opens your link in a chat These are not bulk crawlers. Blocking them makes links that people paste into ChatGPT or Claude fail to open. | Allow | User-agent: ChatGPT-User
Allow: / | |
Model training and evaluation datasets Blocking excludes your content from training. ChatGPT and Claude search answers are unaffected either way — those run on the separate search crawlers above. | Your call | User-agent: GPTBot
Disallow: / # opt out
# or: Allow: / to stay included | |
Controls tokens for Gemini/Vertex AI and Apple Intelligence Denying them stops your content being used by those AI features. Google Search, AI Overviews and Apple web search follow their own crawler rules instead. | Your call | User-agent: Google-Extended
Disallow: / | |
| Everything else, including crawlers you have never heard of A wildcard group is where new AI crawlers land when they have no dedicated rule. Keep it open for public content and disallow only private, app or API paths. | Allow public paths | User-agent: *
Allow: /
Disallow: /account/
Disallow: /admin/ |
Open https://example.com/robots.txt in a private window. Read the named AI groups and the wildcard group — the wildcard one is what most people forget.
Separate search crawlers (citations), training crawlers (datasets), real-time fetchers (user link opens) and controls tokens. One decision never covers all four.
Write a group for each crawler you care about instead of hoping the wildcard group does what you meant.
Bot-management and challenge rules can return 403 or 429 to AI crawlers even when robots.txt allows them.
A page can be fetched but noindexed. That is a different decision from blocking the crawler, and it must be set in HTML or headers, not in robots.txt.
Access gets you fetched. An llms.txt file and clean schema get you understood and quoted — generate one with the llms.txt Generator.
Run the AI visibility scan again, then look for real 200 responses from GPTBot, ClaudeBot and PerplexityBot in your logs.
AI crawler access is whether the bots behind AI search and assistants — GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Bingbot and the rest — are permitted to fetch your pages. It is decided in robots.txt, and can be overridden by your CDN, firewall or a noindex tag. If a crawler cannot fetch a page, that page cannot be cited in an AI answer.
No. GPTBot is OpenAI's training crawler. ChatGPT search results come from OAI-SearchBot, and links a user asks ChatGPT to open are fetched by ChatGPT-User. Blocking GPTBot only removes your content from training data; blocking OAI-SearchBot or ChatGPT-User is what removes your citations and link opens.
No. The scan reads public files and markup: it checks whether GPTBot, ClaudeBot and PerplexityBot are allowed by robots.txt, whether an llms.txt file exists, and whether schema and entity signals are present. It does not ask ChatGPT, Perplexity or Claude anything, so it cannot tell you whether you currently appear in their answers — for that you have to check those tools yourself.
It is a protocol that compliant crawlers follow, and it is the mechanism every major AI vendor documents for opting out. It does not stop clients that ignore it, and it cannot protect private data. Use it for policy — what well-behaved crawlers may read — and authentication plus rate limits for security.
robots.txt controls access: what a crawler may fetch. llms.txt is positive, curated guidance: a markdown map of the pages, docs and policies you want AI systems to read and cite. They complement each other — robots.txt gets you crawled, llms.txt gets you pointed at the right pages.
No. Google Search and AI Overviews follow Googlebot rules, exactly like ordinary rankings. Google-Extended is a separate robots.txt token that controls whether Google may use your content in Gemini apps and Vertex AI, independently of Search.
Scan first — it takes about ten seconds and tells you which of your AI crawler rules are actually in place. Then fix what it flags and re-check.
Last reviewed: October 12, 2026. Example rules are generic — always adapt paths and user-agent tokens to your own site and to each vendor's published crawler list.