ClaudeBot: What It Does and How to Control It
Anthropic's crawler that gathers public web data for training and evaluating its Claude models. Anthropic identifies it through its user-agent token and controls it via robots.txt.
What happens if you allow or block it
Your pages may be included in the vendor's training datasets.
your content is excluded from Anthropic's training datasets — Claude's answers are unaffected.
The robots.txt rule
This is the exact rule auditme.dev publishes for ClaudeBot:
User-agent: ClaudeBot Allow: / Disallow: /auth/ /_next/ /private/ /dashboard /settings /profile Disallow: ... (25 more paths)
The full path list lives in our robots.txt.
Frequently Asked Questions
Is ClaudeBot allowed on auditme.dev?
Yes. Our robots.txt explicitly allows ClaudeBot to crawl public pages while keeping private areas (auth, dashboard, internal APIs) disallowed.
Does blocking ClaudeBot hurt Google rankings?
No. ClaudeBot is Anthropic's crawler and works independently from Googlebot. Blocking it never changes your Google rankings — it only changes your content is excluded from Anthropic's training datasets — Claude's answers are unaffected.
How do I allow or block ClaudeBot on my own site?
Add a "User-agent: ClaudeBot" block to your robots.txt with Allow or Disallow rules, then verify the file parses correctly.