GoogleAgent-URLContext
Fetches pages that Gemini API developers pass as URL context. A user-triggered fetcher, so robots.txt rules generally aren't honored.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | URL-context grounding for Gemini API apps |
| Purpose | User-triggered fetch |
| User-agent token | GoogleAgent-URLContext |
| Respects robots.txt | No (user-triggered) |
GoogleAgent-URLContext retrieves web content when a Gemini API application uses the URL-context tool - developers supply URLs and the fetcher pulls a live copy so the model can ground its answer in the page and cite it.
Like Google's other user-triggered fetchers, Google documents that it generally ignores robots.txt because each fetch is requested by a person or their application. Server- or firewall-level user-agent filtering is the only way to enforce a block.
Allow GoogleAgent-URLContext
Apps built on the Gemini API can ground answers in your real, current pages - your content reaches AI products beyond Google's own.
User-agent: GoogleAgent-URLContext Allow: /
Block GoogleAgent-URLContext
You don't want third-party Gemini-powered apps ingesting your pages - enforce it in your WAF, since robots.txt won't be honored.
User-agent: GoogleAgent-URLContext Disallow: /
Heads up: GoogleAgent-URLContext does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
Can GoogleAgent-URLContext read your page right now?
Test any URL and see exactly what AI crawlers receive.