Google-Agent
Google's agentic fetcher - Gemini navigating and acting on a user's behalf. Documented to ignore robots.txt.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | Gemini agentic browsing and task execution |
| Purpose | User-triggered fetch |
| User-agent token | Google-Agent |
| Respects robots.txt | No (user-triggered) |
Google-Agent identifies fetches made when an AI agent on Google infrastructure navigates the web and performs actions a user requested. Google's own phrasing - navigating and performing user-requested actions - is what separates it from the rest of this directory: this is not a bot reading pages for an index, it is software using your site.
That distinction has consequences beyond content policy. An agent following a multi-step task can move through navigation, forms, and interactive flows rather than fetching one URL and leaving. Rate limits, bot-detection heuristics, and checkout or signup funnels built on the assumption of a human at a keyboard are all touched by this traffic in ways a normal crawler never reaches.
Google documents it among the user-triggered fetchers, which "generally ignore robots.txt rules" because a person initiated the request. Server- or firewall-level user-agent filtering is the only mechanism that actually enforces a block.
Commercially, agentic traffic is the category most likely to carry intent - an agent completing a task on someone's behalf is closer to a customer than to a crawler. Blocking it protects you from automated interaction you did not design for; it also blocks a would-be buyer whose agent was sent to your site.
How Google-Agent behaves
- Navigates and performs actions rather than only reading - can traverse forms and multi-step flows.
- Documented among Google's user-triggered fetchers, which generally ignore robots.txt.
- Sends mobile and desktop user-agent variants carrying the Google-Agent identifier.
- May generate several sequential requests per task, unlike single-page fetchers.
Allow Google-Agent
Gemini users whose tasks touch your site get real content and working interactions - and agentic visits often carry direct commercial intent.
User-agent: Google-Agent Allow: /
Block Google-Agent
You want no automated agents interacting with your site - enforce it in your WAF, since robots.txt is documented as generally ignored.
User-agent: Google-Agent Disallow: /
Heads up: Google-Agent does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
How to verify Google-Agent
Published IP range feed
Verify against Google's user-triggered-fetchers IP feed, which is separate from the Googlebot and common-crawlers files. Verification matters more for agentic traffic than for ordinary crawls: because this token can drive forms and multi-step flows rather than just read pages, treating an unverified request as a legitimate Google agent means granting interactive access on the strength of a header anyone can set.
Documented reverse-DNS suffixes: .google.com
Commonly confused with Google-Agent
Similar naming, different job: URL-context fetches specific URLs a developer supplies to the Gemini API, while Google-Agent navigates to complete a task.
Deep Research gathers sources for a report; Google-Agent takes actions. Both are Gemini-driven and both ignore robots.txt.
Meta's agentic fetcher, with the same weak robots.txt guarantee and the same interactive-traffic implications.
Google-Agent FAQs
How is Google-Agent different from a crawler?
Crawlers read pages for an index. Google-Agent navigates and performs actions a user asked for, so it can move through forms and multi-step flows rather than fetching a single URL - which makes it relevant to your bot detection and funnels, not just your content policy.
Can robots.txt block Google-Agent?
Not reliably. Google documents its user-triggered fetchers as generally ignoring robots.txt because a person initiated the request. Enforcement requires user-agent filtering at the server, CDN, or WAF level.
Should I be worried about agents hitting my forms?
It is worth testing rather than worrying. Rate limits, bot-detection rules, and signup or checkout funnels built for human timing can behave unexpectedly under agentic traffic - a good reason to verify how your flows respond.
Is blocking agentic traffic a good idea commercially?
Consider what you are turning away. An agent completing a task for someone is closer to a prospective customer than to a crawler, so a block protects you from unplanned automation at the cost of a potential buyer's proxy.
Can Google-Agent read your page right now?
Test any URL and see exactly what AI crawlers receive.