ClaudeBot
Anthropic's training crawler for Claude. One of the few AI crawlers that honors Crawl-delay.
Reference reviewed by Sona
| Operator | Anthropic |
|---|---|
| Powers | Training data for Anthropic's Claude models |
| Purpose | Model training |
| User-agent token | ClaudeBot |
| Respects robots.txt | Yes |
ClaudeBot collects publicly available content used in training Anthropic's Claude models. Anthropic's description is that it "helps enhance the utility and safety of our generative AI models by collecting web content" - the Claude counterpart to OpenAI's GPTBot.
Its distinguishing feature among AI training crawlers is throttling. Anthropic documents support for the Crawl-delay directive, an extension most AI crawlers ignore outright. That gives you a middle option the other training bots don't offer: instead of choosing between full access and a hard block, you can set Crawl-delay: 1 and stay in the training set while capping the request rate.
ClaudeBot honors robots.txt, and Anthropic's guidance is to write rules against the exact token. Anthropic documents three bots - ClaudeBot for training, Claude-User for user-requested web access, and Claude-SearchBot for search quality - each with its own group, so a training opt-out here does not affect whether Claude can cite you.
Anthropic has retired earlier identities. The anthropic-ai and Claude-Web tokens circulated in older robots.txt templates and are no longer the bots doing the crawling. Leaving those legacy rules in place is harmless but does nothing; the three current tokens are what you need rules for.
How ClaudeBot behaves
- Honors the Crawl-delay directive - unusual among AI training crawlers and the reason a rate cap is a real alternative to a block here.
- Bulk crawler: broad coverage across your URL space rather than single-page hits.
- Shares Anthropic's published IP range list with Claude-User and Claude-SearchBot, so IP alone will not tell you which bot called.
- Legacy anthropic-ai and Claude-Web tokens are retired; rules naming them no longer match live traffic.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; [email protected])
Allow ClaudeBot
Helps Claude describe your domain, products, and expertise accurately from trained knowledge - and you can cap the crawl rate with Crawl-delay rather than blocking outright.
User-agent: ClaudeBot Allow: /
Block ClaudeBot
You want your content kept out of Claude's training data, and you accept that this is separate from Claude's ability to cite you via search.
User-agent: ClaudeBot Disallow: /
How to verify ClaudeBot
Published IP range feed
Anthropic publishes a single IP list covering its crawler family at claude.com/crawling/bots.json - one feed for ClaudeBot, Claude-User, and Claude-SearchBot rather than a file per bot, so pair the IP check with the user-agent token to tell them apart. Anthropic's framing is direct: an IP on that list indicates the crawler is coming from Anthropic. There is no documented reverse-DNS hostname.
Check an IP against this botCommonly confused with ClaudeBot
ClaudeBot feeds training; Claude-SearchBot feeds the index that lets Claude cite and link you. Blocking training does not remove you from Claude's search results.
ClaudeBot crawls on Anthropic's schedule; Claude-User fetches a page because someone in a Claude conversation asked for it.
The same role at OpenAI, with one practical difference: ClaudeBot honors Crawl-delay and GPTBot does not.
ClaudeBot FAQs
Can I slow ClaudeBot down instead of blocking it?
Yes. Anthropic documents support for Crawl-delay, so a ClaudeBot group with Crawl-delay: 1 throttles the crawl while keeping your content eligible. Most AI training crawlers ignore that directive, which makes this a genuine option unique to ClaudeBot.
Do I still need rules for anthropic-ai or Claude-Web?
No. Those tokens are retired and no longer identify live Anthropic traffic. Write rules for ClaudeBot, Claude-User, and Claude-SearchBot instead. Old rules can stay, but they do nothing.
Does blocking ClaudeBot stop Claude from citing my site?
No. Citation eligibility rides on Claude-SearchBot, and live page reads on Claude-User. ClaudeBot governs training only, so you can opt out of training and remain fully citable.
How do I verify ClaudeBot traffic?
Check the source IP against claude.com/crawling/bots.json. That list covers all of Anthropic's crawlers together, so combine it with the ClaudeBot user-agent token to distinguish training crawls from Claude-User or Claude-SearchBot requests.
Can ClaudeBot read your page right now?
Test any URL and see exactly what AI crawlers receive.