Skip to content
robots.txt

How to Unblock AI Crawlers on Your Website

If GPTBot, ClaudeBot or PerplexityBot suddenly stopped fetching your pages, the cause is almost always one line in robots.txt — usually a rule written for something else and never narrowed down. Here is how to find it, allow each crawler safely, and confirm the fix.

Why AI crawlers get blocked

Most sites never intended to block AI crawlers. A single Disallow: / under User-agent: *, added once while dealing with a scraper or a monitoring bot, silences every crawler at once — including the ones you want reading your site. The same happens with a copied hardening recipe that disallows /bot paths and happens to catch GPTBot, or with a firewall rule that drops requests matching a broad user-agent pattern. WAF and security plugins are a second, invisible layer: robots.txt is a request for cooperation, while a firewall actively refuses the connection, so a crawler can be blocked even when robots.txt looks perfect. The fix is the same in both cases — narrow the rule down to the private paths you actually need protected, and let public pages through.

How to check your robots.txt

Open https://yourdomain.com/robots.txt in the browser and look for a blanket Disallow: / under a wildcard or a named user-agent. Then paste the file into the free robots.txt checker, which flags missing, conflicting and over-broad rules — including whether each AI crawler is allowed or blocked. If the file looks fine but crawlers still fail, the robots.txt guide explains how directives are matched, why order matters, and what each directive does and does not control.

How to allow each crawler

Each crawler has its own user-agent token, so each one needs its own block. Find yours below, then copy the rule from the example further down the page. Every row links to a page explaining what that crawler does, what blocking it costs you and the exact rule that controls it.

User-agentVendorTypeDetails
GPTBotOpenAIAI training/bots/gptbot
OAI-SearchBotOpenAISearch & answers/bots/oai-searchbot
ChatGPT-UserOpenAIReal-time user fetch/bots/chatgpt-user
ClaudeBotAnthropicAI training/bots/claudebot
Claude-SearchBotAnthropicSearch & answers/bots/claude-searchbot
Claude-UserAnthropicReal-time user fetch/bots/claude-user
PerplexityBotPerplexitySearch & answers/bots/perplexitybot
Google-ExtendedGoogleAI controls token/bots/google-extended
Google-CloudVertexBotGoogle CloudSearch & answers/bots/google-cloudvertexbot
BingbotMicrosoftSearch & answers/bots/bingbot
BingPreviewMicrosoftSearch & answers/bots/bingpreview
Applebot-ExtendedAppleAI controls token/bots/applebot-extended

The full directory with allow/block guidance for all twelve crawlers lives on the AI bots directory.

A safe robots.txt example

Allow public pages, and keep the private parts of your site disallowed: account areas, admin panels, carts and internal API endpoints. Repeat the user-agent block for every crawler you want to allow from the table above.

User-agent: GPTBot
Allow: /
Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/

User-agent: ClaudeBot
Allow: /
Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/

User-agent: PerplexityBot
Allow: /
Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/

User-agent: *
Allow: /
Disallow: /account/ /admin/ /cart/ /checkout/ /internal-api/

Sitemap: https://www.example.com/sitemap.xml

Two things to avoid: putting Disallow: / under User-agent: * unless you mean to block everything, and copying allow rules for private application paths — a crawler that can reach your dashboard can also reach your customers' data.

How to verify after changes

Robots.txt edits take effect quickly, but only if the file still parses. Re-open your live /robots.txt URL to confirm the new text is being served, then run it through the robots.txt checker once more — it re-reads the live file and confirms every crawler in the table above is allowed while your private paths stay disallowed. After that, the crawler itself decides when to return: allow rules take effect on its next scheduled visit, not instantly.

Frequently Asked Questions

Will unblocking GPTBot hurt my Google rankings?

No. Googlebot is a separate user-agent with its own robots.txt rules, and Google ignores directives written for other crawlers. Allowing GPTBot changes nothing about how Google indexes or ranks your site.

What is the difference between GPTBot and ChatGPT-User?

GPTBot is a bulk crawler that collects publicly available pages, while ChatGPT-User fetches a single page on behalf of one person who asked for it — for example when ChatGPT opens a link you pasted in. OAI-SearchBot is the third one: it indexes pages for ChatGPT search answers.

How do I block a crawler again?

Delete the Allow line under that crawler's user-agent block and add Disallow rules instead, then re-run the robots.txt checker. Every crawler, what it does and the exact rule that controls it is listed in the bots directory.

Does auditme.dev allow AI crawlers?

Yes. Its robots.txt allows every crawler in the directory to reach public pages while keeping private areas — auth, dashboard, settings and internal API paths — disallowed, which is the pattern shown below.