Google-CloudVertexBot
Crawls sites for Vertex AI Agents. Google documents it as crawling at the site owner's own request.
Reference reviewed by Sona
| Operator | |
|---|---|
| Powers | Vertex AI Agents site ingestion |
| Purpose | User-triggered fetch |
| User-agent token | Google-CloudVertexBot |
| Respects robots.txt | Yes |
Google-CloudVertexBot fetches site content for Vertex AI Agents, and the exact wording of Google's documentation matters here: it "affect crawls requested by the site owners' for building Vertex AI Agents." The intended flow is that you ask for your own site to be ingested, not that an arbitrary third party points a crawler at you.
That framing makes this the least adversarial bot in the directory. If nobody has configured a Vertex AI agent against your domain, you should see no traffic from it at all - so an unexpected Google-CloudVertexBot hit is worth investigating as a configuration question rather than treated as background crawl noise.
It honors robots.txt for its own token, and blocking it has no effect on Googlebot, Google Search rankings, or AI Overviews eligibility. It is a Google Cloud product crawler that happens to run on Google infrastructure, not part of the Search pipeline.
The judgment call is about your partners rather than about AI in general. If customers, resellers, or internal teams build Vertex AI agents that need to answer questions from your documentation, this token is what lets them ground those agents in your real content. If nobody does, a Disallow costs you nothing.
How Google-CloudVertexBot behaves
- Should generate no traffic at all unless someone has configured a Vertex AI agent against your site.
- Honors robots.txt for its own token, independently of Googlebot.
- Blocking it has no effect on Google Search inclusion, rankings, or AI Overviews.
- Covered by Google's common-crawlers IP feed, not the Googlebot-specific one.
Full user-agent string
Mozilla/5.0 (compatible; Google-CloudVertexBot/1.0; +https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers)
Allow Google-CloudVertexBot
Teams building agents on Google Cloud - possibly your own, possibly a customer's - can ground them in your real content.
User-agent: Google-CloudVertexBot Allow: /
Block Google-CloudVertexBot
You don't want third-party Vertex AI applications ingesting your site, and it costs you nothing in Google Search.
User-agent: Google-CloudVertexBot Disallow: /
How to verify Google-CloudVertexBot
Published IP range feed
This crawler is covered by Google's common-crawlers IP range feed rather than by the Googlebot-specific file, so verify against the common-crawlers JSON. Reverse DNS under google.com applies as it does for Google's other infrastructure. Because traffic should only exist when someone has configured a Vertex AI agent against your domain, verification here is often less about spotting forgeries and more about confirming who set that up.
Documented reverse-DNS suffixes: .google.com
Commonly confused with Google-CloudVertexBot
The direct AWS equivalent - a cloud provider's crawler that ingests sites into customer-built knowledge bases. Same decision, different vendor.
Entirely separate pipelines. Blocking Google-CloudVertexBot does not affect Search; blocking Googlebot does.
Both serve Gemini-based developer products, but URL-context fetches are live per-request grounding, while this bot performs configured site ingestion.
Google-CloudVertexBot FAQs
Does blocking Google-CloudVertexBot affect my Google Search rankings?
No. It is a Google Cloud product crawler, entirely separate from the Search pipeline. Blocking it has no effect on Googlebot, your rankings, or AI Overviews eligibility.
Why am I seeing this crawler at all?
Google documents it as serving crawls requested by site owners for building Vertex AI Agents, so traffic normally implies someone configured an agent against your domain. Unexpected hits are worth tracing to whoever set that up.
Is this the crawler that trains Gemini?
No. Gemini training use is governed by the Google-Extended token. This bot ingests site content into specific customer-built Vertex AI agents.
Should I allow it?
It depends on whether anyone builds agents against your content. If your own teams, customers, or partners run Vertex AI agents that need your documentation, allow it. If not, a Disallow costs you nothing.
Can Google-CloudVertexBot read your page right now?
Test any URL and see exactly what AI crawlers receive.