Google-Extended: What It Does and How to Control It
Not a standalone search crawler — it is a controls token you add to robots.txt to manage whether Google may use your content for Gemini apps and Vertex AI grounding, independently of Google Search. Appearing in Google Search (and AI Overviews) follows Googlebot rules, not Google-Extended.
What happens if you allow or block it
The vendor's AI features may use your content according to the vendor's default policy.
Gemini apps and Vertex AI stop using your content, while Google Search indexing and AI Overviews keep following your Googlebot rules.
The robots.txt rule
This is the exact rule auditme.dev publishes for Google-Extended:
User-agent: Google-Extended Allow: / Disallow: /auth/ /_next/ /private/ /dashboard /settings /profile Disallow: ... (25 more paths)
The full path list lives in our robots.txt.
Frequently Asked Questions
Is Google-Extended allowed on auditme.dev?
Yes. Our robots.txt explicitly allows Google-Extended to crawl public pages while keeping private areas (auth, dashboard, internal APIs) disallowed.
Does blocking Google-Extended hurt Google rankings?
No. Google-Extended is Google's crawler and works independently from Googlebot. Blocking it never changes your Google rankings — it only changes Gemini apps and Vertex AI stop using your content, while Google Search indexing and AI Overviews keep following your Googlebot rules.
How do I allow or block Google-Extended on my own site?
Add a "User-agent: Google-Extended" block to your robots.txt with Allow or Disallow rules, then verify the file parses correctly.