Googlebot
Google's main search crawler - and the index behind AI Overviews. There is no separate AI Overviews bot.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | Google Search indexing and AI Overviews surfacing |
| Purpose | Search & answers |
| User-agent token | Googlebot |
| Respects robots.txt | Yes |
Googlebot is the crawler behind Google Search, covering Search, Discover, Images, Video, and News. It earns a place in an AI-crawler directory for one reason: AI Overviews are generated from the index Googlebot builds. There is no AI Overviews crawler to allow or block, which means your AI visibility inside Google is decided by ordinary technical SEO rather than by any AI-specific rule.
That collapses a decision people expect to be able to make. You cannot be in Google Search but out of AI Overviews via robots.txt - the access is the same access. The available controls are content-level rather than crawler-level: nosnippet, max-snippet, and data-nosnippet limit what can be quoted, and they apply to Overviews as they do to featured snippets.
Googlebot is also the most rendering-capable crawler in this directory by a wide margin. It executes JavaScript through a headless Chromium that tracks recent stable Chrome, so client-side-rendered content that AI training crawlers miss entirely is generally still indexed here. If a page renders in Search but is invisible to GPTBot or ClaudeBot, this is usually why.
Blocking Googlebot removes you from Google Search and from AI Overviews simultaneously, and is almost never the right call. The AI-training question at Google is handled by the separate Google-Extended token, which Google documents as having no effect on Search inclusion or rankings.
How Googlebot behaves
- Renders JavaScript via an evergreen headless Chromium, so client-side content is usually indexed - unlike most AI training crawlers.
- Ignores Crawl-delay entirely; crawl rate is managed by Google's own scheduling and Search Console settings.
- Falls back to the Googlebot group for Googlebot-Image and Googlebot-News unless those tokens have their own rules.
- Its index feeds AI Overviews directly, so there is no separate AI crawl to permit or refuse.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html)
Allow Googlebot
You stay in Google Search and remain eligible for AI Overviews - for nearly every site this is not a real decision.
User-agent: Googlebot Allow: /
Block Googlebot
Almost never advisable: this removes you from Google Search itself, not just its AI features. Use Google-Extended if training is your concern.
User-agent: Googlebot Disallow: /
How to verify Googlebot
Reverse DNS + published IP ranges
Googlebot supports the strongest verification of any crawler here, and it is worth doing properly: run a reverse DNS lookup on the request IP, confirm the hostname ends in googlebot.com or google.com, then run a forward lookup on that hostname and confirm it returns the original IP. The second step is the one people skip, and it is the step that matters - reverse DNS alone can be controlled by whoever owns the address block. Google also publishes the ranges as JSON if you prefer a list to two DNS round-trips.
Documented reverse-DNS suffixes: .googlebot.com, .google.com
Commonly confused with Googlebot
Googlebot controls Search access; Google-Extended only controls whether the result may train Gemini. Blocking Googlebot to avoid AI training is a costly mistake.
GoogleOther is a separate generic crawler for non-Search product teams. Blocking it does not affect Search; blocking Googlebot does.
The same structural situation at Microsoft, where the Bing index feeds Copilot with no separate Copilot crawler.
Googlebot FAQs
How do I opt out of AI Overviews?
There is no crawler-level opt-out, because AI Overviews use the standard Search index. Your options are content-level: the nosnippet or max-snippet robots meta tags, or data-nosnippet on specific elements. Blocking Googlebot would work but removes you from Search entirely.
Is there a separate Google AI crawler I should know about?
Not for AI Overviews - that runs on Googlebot's index. Google does operate other AI-adjacent bots: Google-Extended as a training control token, GoogleOther for product-team crawls, Google-CloudVertexBot for site-owner-requested Vertex AI ingestion, and several user-triggered fetchers.
Why does my page show in Google but not in ChatGPT or Claude?
Googlebot renders JavaScript; most AI training and search crawlers do not, or do so far less reliably. If your content only appears after client-side hydration, Google sees it and the AI crawlers largely do not.
Does Googlebot respect Crawl-delay?
No. Google ignores the Crawl-delay directive. Crawl rate is set by Google's own scheduling, with limited influence available through Search Console rather than robots.txt.
Can Googlebot read your page right now?
Test any URL and see exactly what AI crawlers receive.