Claude-SearchBot
Anthropic's search crawler. Controls whether your pages can be cited when Claude searches the web.
Reference reviewed by Sona
| Operator | Anthropic |
|---|---|
| Powers | Claude's search index and citations |
| Purpose | Search & answers |
| User-agent token | Claude-SearchBot |
| Respects robots.txt | Yes |
Claude-SearchBot exists to improve search quality - Anthropic's wording is that it "navigates the web to improve search result quality for users." It builds the retrieval layer Claude draws on when it searches, which makes this the token that governs whether Claude can put your page in front of someone with a link attached.
It is the newest of Anthropic's three documented bots and the one most often missing from robots.txt files written a year or two ago. Those files typically name ClaudeBot and stop, which means sites that intended to opt out of training are frequently either silently allowing search crawling or, if they wrote a broad wildcard Disallow, silently blocking their own citations.
Anthropic states it honors industry-standard robots.txt directives, and it reads its own group. The split is deliberate and worth using: Disallow ClaudeBot to stay out of training, Allow Claude-SearchBot to stay citable. Nothing about that combination is contradictory or unsupported.
Because its job is retrieval rather than collection, the same practical constraints apply as with any search crawler. It needs a 200, content in the server-rendered HTML, and no gate in front of the substance. An allow rule makes you eligible; the page still has to be readable.
How Claude-SearchBot behaves
- Revisits pages to keep the search index fresh, so expect recurring rather than one-off traffic.
- The most commonly omitted Anthropic token in existing robots.txt files, which makes it a frequent source of unintended blocks and unintended allows.
- Shares Anthropic's IP feed with ClaudeBot and Claude-User - distinguish by user-agent token, not by address.
- Needs content in the server-rendered HTML to index a page for citation.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; [email protected])
Allow Claude-SearchBot
Your pages stay eligible to appear as cited, linked sources when Claude searches the web on a user's behalf.
User-agent: Claude-SearchBot Allow: /
Block Claude-SearchBot
You don't want your content surfaced inside Claude's answers, accepting that you lose citation referrals as well as visibility.
User-agent: Claude-SearchBot Disallow: /
How to verify Claude-SearchBot
Published IP range feed
Verify against Anthropic's shared crawler list at claude.com/crawling/bots.json. Since one feed covers ClaudeBot, Claude-User, and Claude-SearchBot, the IP check proves the traffic is Anthropic's but not which bot it is - read the user-agent token for that. If you are auditing whether a training block is working, the token is the field that tells you whether you are looking at a training crawl or a search crawl.
Check an IP against this botCommonly confused with Claude-SearchBot
The pair most often conflated. ClaudeBot is training, Claude-SearchBot is citation. A wildcard Disallow intended for training takes out citations too.
Claude-SearchBot indexes in advance; Claude-User fetches live during a conversation. Both feed Claude's answers, at different moments.
OpenAI's direct equivalent. If you are setting a citation policy, these two tokens should almost always get the same treatment.
Claude-SearchBot FAQs
Is Claude-SearchBot the same as ClaudeBot?
No, and the difference decides whether Claude can link to you. ClaudeBot collects training data; Claude-SearchBot builds the search index behind Claude's citations. They are separate robots.txt groups.
My robots.txt only mentions ClaudeBot. What am I missing?
Claude-SearchBot and Claude-User have no rules, so they follow whatever your wildcard group says. If that group is a broad Disallow, you are blocking your own Claude citations without intending to.
Can I stay out of Claude's training but still be cited by Claude?
Yes. Disallow ClaudeBot and allow Claude-SearchBot. Anthropic reads the groups independently, so this is a supported configuration rather than a workaround.
Does Claude-SearchBot honor Crawl-delay like ClaudeBot?
Anthropic documents Crawl-delay support in the context of its crawling guidance, but the explicit throttling example is given for ClaudeBot. Treat rate limiting as reliable for ClaudeBot and test before depending on it here.
Can Claude-SearchBot read your page right now?
Test any URL and see exactly what AI crawlers receive.