Google-Extended
A robots.txt token, not a crawler. Controls Gemini training use without touching Google Search.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | Gemini and Vertex AI generative training |
| Purpose | Training opt-out token |
| User-agent token | Google-Extended |
| Respects robots.txt | Yes |
Google-Extended sends no traffic. There is no Google-Extended user-agent in your logs and never will be, because it is not a bot - it is a control token that tells Google whether content Googlebot already fetched may be used to train generative models. You are annotating an existing crawl, not permitting a new one.
Google's documentation is unusually explicit about the consequences, and the wording is worth keeping to hand when someone objects: disallowing Google-Extended "does not impact a site's inclusion in Google Search nor is it used as a ranking signal." This is the answer to the most common fear about opting out of AI training at Google.
What it does not cover is AI Overviews. Those are generated from the regular Search index that Googlebot builds, so disallowing Google-Extended does not remove you from AI Overviews - and there is no separate token that does. The only way out of AI Overviews is to leave Search itself, or to use the nosnippet and max-snippet robots meta directives to restrict what can be shown.
Because the token controls usage rather than access, it is one of the few entries in this directory where a Disallow costs you nothing measurable in traffic. Sites that want a defensible "we do not license our content for AI training" position while keeping full search visibility generally start here.
How Google-Extended behaves
- Generates zero requests - it annotates Googlebot's existing crawl rather than performing its own.
- Disallowing it does not remove you from Google Search and is not a ranking signal, per Google's documentation.
- Does not control AI Overviews, which draw on the standard Search index built by Googlebot.
- Any log line claiming a Google-Extended user-agent is spoofed, since the real token never sends one.
Allow Google-Extended
Your content can be used to train and ground Gemini, shaping how Google's models describe your domain from knowledge rather than retrieval.
User-agent: Google-Extended Allow: /
Block Google-Extended
You want out of generative-AI training while keeping full Google Search visibility - Google documents that this costs you no rankings and no inclusion.
User-agent: Google-Extended Disallow: /
How to verify Google-Extended
No published verification method
There is nothing to verify. Google-Extended has no user-agent and generates no requests, so any traffic claiming to be Google-Extended is fabricated by definition. If you want to confirm Google is honoring the directive, you cannot do it from your logs - test it from the output side by checking whether Gemini reproduces content only your site publishes.
Check an IP against this botCommonly confused with Google-Extended
Googlebot does the actual fetching and controls Search inclusion. Google-Extended only labels how the already-fetched content may be used.
Apple's identical pattern - a usage-control token layered over a real crawler (Applebot), not a crawler itself.
GoogleOther is a real crawler with real traffic used by various Google product teams. Google-Extended is a policy flag with no traffic at all.
Google-Extended FAQs
Will blocking Google-Extended hurt my Google rankings?
No. Google's documentation states that Google-Extended does not impact a site's inclusion in Google Search and is not used as a ranking signal. It governs generative-AI training use only.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews are generated from the regular Search index that Googlebot builds, and there is no token that opts out of them specifically. Your options are the nosnippet or max-snippet meta directives, or leaving Search entirely.
Why do I never see Google-Extended in my server logs?
Because it is not a crawler. It sends no requests and has no user-agent string - it exists purely as a robots.txt directive that tells Google how content Googlebot already collected may be used.
What is the correct robots.txt rule?
A standalone group: User-agent: Google-Extended followed by Disallow: /. It needs its own group rather than being folded into a Googlebot rule, since the two tokens control entirely different things.
Can Google-Extended read your page right now?
Test any URL and see exactly what AI crawlers receive.