Claude-SearchBot: What It Does and How to Control It
Anthropic's crawler for improving and grounding the sources Claude surfaces when it answers questions with web results.
What happens if you allow or block it
Your pages can be indexed and cited in the vendor's search and AI answers.
your pages are much less likely to be cited as a source in Claude's answers.
The robots.txt rule
This is the exact rule auditme.dev publishes for Claude-SearchBot:
User-agent: Claude-SearchBot Allow: / Disallow: /auth/ /_next/ /private/ /dashboard /settings /profile Disallow: ... (25 more paths)
The full path list lives in our robots.txt.
Frequently Asked Questions
Is Claude-SearchBot allowed on auditme.dev?
Yes. Our robots.txt explicitly allows Claude-SearchBot to crawl public pages while keeping private areas (auth, dashboard, internal APIs) disallowed.
Does blocking Claude-SearchBot hurt Google rankings?
No. Claude-SearchBot is Anthropic's crawler and works independently from Googlebot. Blocking it never changes your Google rankings — it only changes your pages are much less likely to be cited as a source in Claude's answers.
How do I allow or block Claude-SearchBot on my own site?
Add a "User-agent: Claude-SearchBot" block to your robots.txt with Allow or Disallow rules, then verify the file parses correctly.