bedrockbot
AWS's crawler for Bedrock knowledge bases. Its user-agent carries a per-crawler UUID, which trips exact-match rules.
Reference reviewed by Sona
| Operator | Amazon (AWS) |
|---|---|
| Powers | Amazon Bedrock knowledge-base crawls |
| Purpose | User-triggered fetch |
| User-agent token | bedrockbot |
| Respects robots.txt | Yes |
bedrockbot crawls URLs that Amazon Bedrock customers configure as web data sources for their knowledge bases - the retrieval layer behind custom RAG applications and enterprise assistants built on AWS. Nothing here is Amazon's own initiative: every fetch traces back to a customer who entered your URL.
AWS documents that the crawler always adheres to robots.txt, honoring both Allow and Disallow directives. That is a stronger commitment than most cloud-ingestion crawlers make, and stronger than the "may not follow" language Amazon uses for its consumer-side Amzn-User fetcher.
There is one implementation detail that catches people out. The live user-agent carries a per-crawler UUID suffix - bedrockbot_UUID - so a robots.txt or WAF rule written against the exact string will not match real traffic. Match on the bedrockbot prefix, and use the bare token in robots.txt where prefix matching is the norm anyway.
Strategically this is the AWS twin of Google-CloudVertexBot: the question is not whether you trust Amazon but whether you want customer- and partner-built AI assistants grounded in your documentation. For anyone selling to enterprises that build on Bedrock, blocking it can quietly break a customer's internal tooling.
How bedrockbot behaves
- User-agent carries a per-crawler UUID suffix (bedrockbot_UUID) - match on the prefix, never the exact string.
- AWS documents that it always adheres to robots.txt, honoring both Allow and Disallow.
- Only crawls URLs a Bedrock customer explicitly configured as a data source.
- Originates from general AWS ranges, so source IP does not distinguish it from other AWS traffic.
Full user-agent string
bedrockbot (+https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html)
Allow bedrockbot
Enterprises building assistants on AWS - quite possibly your own customers - can ground them in your real documentation.
User-agent: bedrockbot Allow: /
Block bedrockbot
You don't want third-party Bedrock applications ingesting your site, and AWS documents that a robots.txt Disallow will be honored.
User-agent: bedrockbot Disallow: /
How to verify bedrockbot
No published verification method
AWS publishes no dedicated IP range feed or reverse-DNS convention for bedrockbot, and because crawls run from customer-triggered AWS infrastructure the source addresses sit in general-purpose AWS ranges shared with everything else on the platform. Confirming a request came from AWS therefore proves very little. The UUID-suffixed user-agent is the most distinctive signal available, and it is not an authentication mechanism.
Check an IP against this botCommonly confused with bedrockbot
The direct Google Cloud equivalent - customer-configured site ingestion for building AI agents. Treat both the same way.
Different sides of Amazon: Amazonbot crawls on Amazon's own initiative and may train models, while bedrockbot only fetches what an AWS customer configured.
Also user-driven, but consumer-side - Amzn-User serves Alexa and Amazon AI, and unlike bedrockbot it may not follow robots.txt.
bedrockbot FAQs
Why doesn't my bedrockbot rule match anything?
Almost certainly the UUID. The live user-agent is bedrockbot_UUID with a per-crawler identifier appended, so any rule pinned to the exact literal string fails. Match on the bedrockbot prefix instead.
Does bedrockbot respect robots.txt?
Yes. AWS documents that the crawler always adheres to robots.txt and honors both Allow and Disallow directives - a firmer commitment than Amazon makes for its consumer-side Amzn-User fetcher.
Who is actually crawling me when I see bedrockbot?
An AWS customer who added your URL as a web data source for a Bedrock knowledge base. It is never Amazon's own initiative, so the traffic implies someone is building an assistant that references your content.
Can I verify bedrockbot traffic?
Not meaningfully. There is no dedicated IP feed or reverse-DNS convention, and crawls come from general AWS ranges shared with all other AWS workloads, so confirming an AWS origin tells you very little.
Can bedrockbot read your page right now?
Test any URL and see exactly what AI crawlers receive.