Google-Agent
Google's agentic fetcher - retrieves pages when Gemini acts on a user's behalf. Documented to ignore robots.txt.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | Gemini Agent agentic fetches |
| Purpose | User-triggered fetch |
| User-agent token | Google-Agent |
| Respects robots.txt | No (user-triggered) |
Google-Agent is the user-agent for fetches made when AI agents on Google infrastructure act for a user - today that means Gemini's agentic features, including Gemini Agent, which absorbed Project Mariner (Google's browser-driving agent, retired in May 2026) and identifies itself with this token.
It appears in Google's user-triggered fetchers documentation, which states these fetchers generally ignore robots.txt because a person initiated the request. Server- or firewall-level user-agent filtering is the only way to enforce a block.
Allow Google-Agent
Gemini users whose tasks touch your site get real content, and agentic visits often carry direct commercial intent.
User-agent: Google-Agent Allow: /
Block Google-Agent
You want no agentic AI traffic at all - enforce it in your WAF, since robots.txt won't be honored.
User-agent: Google-Agent Disallow: /
Heads up: Google-Agent does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
Can Google-Agent read your page right now?
Test any URL and see exactly what AI crawlers receive.