How to Unblock AI Crawlers on Your Website
If GPTBot, ClaudeBot or PerplexityBot suddenly stopped fetching your pages, the cause is almost always one line in robots.txt — usually a rule written for something else and never narrowed down. Here is how to find it, allow each crawler safely, and confirm the fix.
Why AI crawlers get blocked
Most sites never intended to block AI crawlers. A single Disallow: / under User-agent: *, added once while dealing with a scraper or a monitoring bot, silences every crawler at once — including the ones you want reading your site. The same happens with a copied hardening recipe that disallows /bot paths and happens to catch GPTBot, or with a firewall rule that drops requests matching a broad user-agent pattern. WAF and security plugins are a second, invisible layer: robots.txt is a request for cooperation, while a firewall actively refuses the connection, so a crawler can be blocked even when robots.txt looks perfect. The fix is the same in both cases — narrow the rule down to the private paths you actually need protected, and let public pages through.
How to check your robots.txt
Open https://yourdomain.com/robots.txt in the browser and look for a blanket Disallow: / under a wildcard or a named user-agent. Then paste the file into the free robots.txt checker, which flags missing, conflicting and over-broad rules — including whether each AI crawler is allowed or blocked. If the file looks fine but crawlers still fail, the robots.txt guide explains how directives are matched, why order matters, and what each directive does and does not control.
How to allow each crawler
Each crawler has its own user-agent token, so each one needs its own block. Find yours below, then copy the rule from the example further down the page. Every row links to a page explaining what that crawler does, what blocking it costs you and the exact rule that controls it.
| User-agent | Vendor | Type | Details |
|---|---|---|---|
| GPTBot | OpenAI | AI training | /bots/gptbot |
| OAI-SearchBot | OpenAI | Search & answers | /bots/oai-searchbot |
| ChatGPT-User | OpenAI | Real-time user fetch | /bots/chatgpt-user |
| ClaudeBot | Anthropic | AI training | /bots/claudebot |
| Claude-SearchBot | Anthropic | Search & answers | /bots/claude-searchbot |
| Claude-User | Anthropic | Real-time user fetch | /bots/claude-user |
| PerplexityBot | Perplexity | Search & answers | /bots/perplexitybot |
| Google-Extended | AI controls token | /bots/google-extended | |
| Google-CloudVertexBot | Google Cloud | Search & answers | /bots/google-cloudvertexbot |
| Bingbot | Microsoft | Search & answers | /bots/bingbot |
| BingPreview | Microsoft | Search & answers | /bots/bingpreview |
| Applebot-Extended | Apple | AI controls token | /bots/applebot-extended |
The full directory with allow/block guidance for all twelve crawlers lives on the AI bots directory.
A safe robots.txt example
Allow public pages, and keep the private parts of your site disallowed: account areas, admin panels, carts and internal API endpoints. Repeat the user-agent block for every crawler you want to allow from the table above.
User-agent: GPTBot Allow: / Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/ User-agent: ClaudeBot Allow: / Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/ User-agent: PerplexityBot Allow: / Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/ User-agent: * Allow: / Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/ Sitemap: https://www.example.com/sitemap.xml
Two things to avoid: putting Disallow: / under User-agent: *unless you mean to block everything, and copying allow rules for private application paths — a crawler that can reach your dashboard can also reach your customers' data.
How to verify after changes
Robots.txt edits take effect quickly, but only if the file still parses. Re-open your live /robots.txt URL to confirm the new text is being served, then run it through the robots.txt checker once more — it re-reads the live file and confirms every crawler in the table above is allowed while your private paths stay disallowed. After that, the crawler itself decides when to return: allow rules take effect on its next scheduled visit, not instantly.
Frequently Asked Questions
Will unblocking GPTBot hurt my Google rankings?
No. Googlebot is a separate user-agent with its own robots.txt rules, and Google ignores directives written for other crawlers. Allowing GPTBot changes nothing about how Google indexes or ranks your site.
What is the difference between GPTBot and ChatGPT-User?
GPTBot is a bulk crawler that collects publicly available pages, while ChatGPT-User fetches a single page on behalf of one person who asked for it — for example when ChatGPT opens a link you pasted in. OAI-SearchBot is the third one: it indexes pages for ChatGPT search answers.
How do I block a crawler again?
Delete the Allow line under that crawler's user-agent block and add Disallow rules instead, then re-run the robots.txt checker. Every crawler, what it does and the exact rule that controls it is listed in the bots directory.
Does auditme.dev allow AI crawlers?
Yes. Its robots.txt allows every crawler in the directory to reach public pages while keeping private areas — auth, dashboard, settings and internal API paths — disallowed, which is the pattern shown below.