Claude-SearchBot
Anthropic's search crawler. Controls whether your pages can surface and be cited when Claude searches the web.
| Operator | Anthropic |
|---|---|
| Powers | Claude's search index and citations |
| Purpose | Search & answers |
| User-agent token | Claude-SearchBot |
| Respects robots.txt | Yes |
Claude-SearchBot indexes pages so they can appear as sources when Claude answers questions using web search. Like OpenAI's OAI-SearchBot, its purpose is discovery and citation rather than model training.
Blocking Claude-SearchBot is independent of blocking ClaudeBot: you can opt out of training (ClaudeBot) while still allowing your pages to be surfaced and linked in Claude's answers, or vice versa.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])
Allow Claude-SearchBot
You want your pages eligible to appear as cited sources when Claude answers a relevant query.
User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
You don't want your pages surfaced inside Claude's answer interface.
User-agent: Claude-SearchBot Disallow: /
Can Claude-SearchBot read your page right now?
Test any URL and see exactly what AI crawlers receive.