What Is robots.txt?
robots.txt is a text file placed at the root of a website (e.g., https://example.com/robots.txt) that tells web crawlers which pages or sections of the site they are allowed or disallowed from accessing. It uses the Robots Exclusion Protocol. Search engines and AI crawlers typically check robots.txt before crawling any page.
Why It Matters
robots.txt controls how search engines and AI crawlers interact with your site. Incorrectly configured robots.txt can block important pages from being indexed, waste crawl budget on unimportant pages, or fail to guide crawlers to your most valuable content. For AI discoverability, robots.txt determines whether AI search engines can access your content.
How AuditMe Detects It
AuditMe fetches and parses your robots.txt file during audits. It checks: (1) presence and accessibility of robots.txt, (2) whether important pages are accidentally blocked, (3) whether AI bots (GPTBot, ClaudeBot, PerplexityBot) are allowed or blocked, (4) consistency between robots.txt and meta robots tags, (5) presence of Sitemap directive pointing to your XML sitemap.
Example
User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Sitemap: https://www.example.com/sitemap.xml
User-agent: * Disallow: / # Blocks ALL crawlers from ALL pages # Your site will disappear from search results
Common Mistakes
- Disallow: / — accidentally blocking the entire site from crawlers
- Blocking CSS/JS files — prevents search engines from rendering pages correctly
- No Sitemap directive — crawlers can't find your sitemap
- Blocking AI bots when you want AI discoverability
- Conflicting rules — Allow and Disallow for the same path
- Forgetting to update after site migration or restructuring