Googlebot
Google's main search crawler — and the bot whose index powers AI Overviews. Blocking it removes you from Google Search entirely.
| Operator | |
|---|---|
| Powers | Google Search indexing and AI Overviews surfacing |
| Purpose | Search & answers |
| User-agent token | Googlebot |
| Respects robots.txt | Yes |
Googlebot is the crawler behind Google Search. It matters in an AI context because AI Overviews — the generated answers at the top of Google results — surface content from the regular search index that Googlebot builds. There is no separate AI Overviews crawler to allow or block.
Blocking Googlebot removes your pages from Google Search and from AI Overviews at the same time. To opt out of Gemini training while staying in Search and AI Overviews, block the Google-Extended token instead and leave Googlebot allowed.
Full user-agent string
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Allow Googlebot
You stay in Google Search and remain eligible to be surfaced in AI Overviews — for most sites this is non-negotiable.
User-agent: Googlebot Allow: /
Block Googlebot
Almost never advisable: blocking Googlebot removes you from Google Search itself, not just AI features.
User-agent: Googlebot Disallow: /
Can Googlebot read your page right now?
Test any URL and see exactly what AI crawlers receive.