GoogleAgent-URLContext
Fetches URLs that Gemini API developers pass as context. Community-tracked; absent from Google's crawler docs.
Reference reviewed by Sona
The operator publishes little or no documentation for this token. Details here come from community crawler registries and observed behavior, not vendor confirmation.
| Operator | |
|---|---|
| Powers | URL-context grounding for Gemini API applications |
| Purpose | User-triggered fetch |
| User-agent token | GoogleAgent-URLContext |
| Respects robots.txt | No (user-triggered) |
GoogleAgent-URLContext retrieves web content when an application built on the Gemini API uses the URL-context tool: a developer supplies URLs, and the fetcher pulls a live copy so the model can ground its answer in the page rather than in trained knowledge.
The important structural point is who is on the other end. This is not Google surfacing your content in a Google product - it is third-party software, built by someone you have no relationship with, reading your pages through Google's infrastructure. Whatever that application does with the content is outside Google's answer interfaces and outside your visibility.
As with Gemini-Deep-Research, the token is community-tracked rather than officially documented. The URL-context tool is documented in Google's Gemini API and Cloud materials, but this user-agent does not appear in Google's crawler documentation, so its robots.txt behavior is inferred from how Google's documented user-triggered fetchers behave - generally ignoring robots.txt because a person or their application initiated the request.
Practically, that puts enforcement at the edge if you want it. The upside to allowing it is real but diffuse: your content reaches AI products built well beyond Google's own surfaces, with live rather than stale information, and often with a citation the developer chose to render.
How GoogleAgent-URLContext behaves
- Fetches specific URLs supplied by a developer through the Gemini API URL-context tool.
- Not listed in Google's crawler documentation - the token is community-tracked.
- The requesting third-party application is invisible to you; only Google's infrastructure is identifiable.
- robots.txt behavior is inferred from Google's documented user-triggered fetchers rather than stated.
Allow GoogleAgent-URLContext
Applications built on the Gemini API can ground answers in your current pages, extending your content's reach beyond Google's own products.
User-agent: GoogleAgent-URLContext Allow: /
Block GoogleAgent-URLContext
You don't want third-party Gemini-powered apps ingesting your pages - enforce it in your WAF, since robots.txt likely won't be honored.
User-agent: GoogleAgent-URLContext Disallow: /
Heads up: GoogleAgent-URLContext does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
How to verify GoogleAgent-URLContext
Published IP range feed
No feed is published for this token specifically, since Google does not document it. Google's user-triggered-fetchers range file is the nearest applicable source and reverse DNS under google.com covers Google infrastructure broadly - but neither authenticates this identifier. A further wrinkle: verifying that traffic is Google's tells you nothing about which third-party application requested the fetch, and that is the part you might actually care about.
Documented reverse-DNS suffixes: .google.com
Commonly confused with GoogleAgent-URLContext
Both serve Google-hosted AI development, but Vertex ingests a site into a configured agent while this fetches individual URLs live per request.
Similar naming, different job: Google-Agent navigates and acts on a user task, and unlike this token it is officially documented.
Also community-tracked and Gemini-related, but Deep Research discovers its own sources rather than fetching URLs a developer supplied.
GoogleAgent-URLContext FAQs
Who is actually reading my page when this fetcher hits it?
A third-party application built on the Gemini API, not a Google product. Google's infrastructure performs the fetch, but the developer who supplied the URL is invisible to you - and so is whatever their app does with the content.
Is this token documented by Google?
No. The URL-context tool is documented in Google's Gemini API and Cloud materials, but this user-agent does not appear in Google's crawler documentation. It is tracked by third-party crawler registries.
Can robots.txt block it?
Probably not. Inferring from Google's documented user-triggered fetchers, which generally ignore robots.txt when a person or their application initiated the request, expect a Disallow to be ineffective and use edge filtering instead.
How is this different from Google-CloudVertexBot?
Vertex ingests a whole site into a configured agent, typically at the site owner's request. This fetches individual URLs live, per request, for any developer using the Gemini API's URL-context tool.
Can GoogleAgent-URLContext read your page right now?
Test any URL and see exactly what AI crawlers receive.