bedrockbot
AWS's crawler for Amazon Bedrock - fetches your site when Bedrock customers add it as a knowledge-base source for their AI apps.
Reference reviewed by Sona
| Operator | Amazon (AWS) |
|---|---|
| Powers | Amazon Bedrock knowledge-base crawls |
| Purpose | User-triggered fetch |
| User-agent token | bedrockbot |
| Respects robots.txt | Yes |
bedrockbot crawls URLs that Amazon Bedrock customers configure as web data sources for their knowledge bases - the retrieval layer behind custom RAG applications and enterprise assistants built on Bedrock.
AWS documents that the crawler always adheres to robots.txt, honoring Allow and Disallow directives. In practice its user-agent carries a per-crawler UUID suffix (bedrockbot_UUID), but robots.txt rules for the generic bedrockbot token apply.
Full user-agent string
bedrockbot (+https://docs.aws.amazon.com/bedrock/latest/userguide/webcrawl-data-source-connector.html)
Allow bedrockbot
Enterprises building AI assistants on AWS can ground them in your content - relevant when partners or customers build on Bedrock.
User-agent: bedrockbot Allow: /
Block bedrockbot
You don't want third-party Bedrock applications ingesting your site.
User-agent: bedrockbot Disallow: /
Can bedrockbot read your page right now?
Test any URL and see exactly what AI crawlers receive.