Claude-User
Fetches a page live when a Claude user asks Claude to read it. Notably, Anthropic says it respects do-not-crawl signals.
Reference reviewed by Sona
| Operator | Anthropic |
|---|---|
| Powers | Live page fetches when Claude acts for a user |
| Purpose | User-triggered fetch |
| User-agent token | Claude-User |
| Respects robots.txt | Yes |
Claude-User fires when a person using Claude requests web access - asking Claude to read, summarize, or check a specific page. Anthropic describes it as supporting Claude users when they request web access through their queries.
What makes it worth a second look is its robots.txt posture. Anthropic states Claude-User respects robots.txt do-not-crawl signals. That puts it in a small minority: the equivalent fetchers at OpenAI, Perplexity, Meta, Google, and Amazon are all documented as possibly or generally ignoring robots.txt because a human initiated the request. Anthropic reaches the opposite conclusion from the same premise.
The practical consequence is that a Disallow here is likely to work as written, which is not something you can assume for user-triggered fetchers generally. It also means an over-broad wildcard Disallow genuinely does stop Claude from reading pages users ask about - a self-inflicted block that the other vendors' fetchers would have ignored.
Anthropic's older Claude-Web and anthropic-ai tokens covered similar access and are retired. Claude-User is the current token for on-demand reads, and it is the one to name if you want to allow human-initiated visits while disallowing ClaudeBot's training crawl.
How Claude-User behaves
- Single-page, on-demand fetches tied to one conversation - not a sweep of your site.
- Documented as respecting robots.txt do-not-crawl signals, unlike most user-triggered fetchers at other vendors.
- Traffic tracks Claude usage patterns, so it appears in unpredictable bursts rather than on a crawl schedule.
- Shares Anthropic's published IP feed with ClaudeBot and Claude-SearchBot.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; [email protected])
Allow Claude-User
A user asking Claude about your page gets an answer grounded in the live page - and unlike most fetchers, your robots.txt preference here is actually honored either way.
User-agent: Claude-User Allow: /
Block Claude-User
You want no programmatic access from Claude at all, including human-initiated reads - and here a robots.txt Disallow is documented to be sufficient.
User-agent: Claude-User Disallow: /
How to verify Claude-User
Published IP range feed
Anthropic's shared feed at claude.com/crawling/bots.json covers Claude-User along with its two siblings. Because Anthropic documents this bot as honoring robots.txt, unexpected Claude-User hits on disallowed paths are a reason to check the IP against the feed - genuine Anthropic traffic should be following your rules, so a request that isn't is more likely a forgery than a policy change.
Check an IP against this botCommonly confused with Claude-User
Same role, opposite documented policy: OpenAI says robots.txt may not apply to ChatGPT-User, while Anthropic says Claude-User respects do-not-crawl signals.
Claude-User reads one page a human asked about; ClaudeBot crawls in bulk for training. Allowing the first while blocking the second is a common posture.
Claude-Code covers fetches from the Claude Code CLI, a developer tool, rather than from Claude's chat interface.
Claude-User FAQs
Does Claude-User actually honor robots.txt?
Anthropic states that it respects robots.txt do-not-crawl signals. That is a stronger commitment than OpenAI, Perplexity, Google, Meta, or Amazon make for their equivalent user-triggered fetchers, all of which are documented as possibly ignoring robots.txt.
Will a wildcard Disallow block Claude from reading pages users ask about?
Yes, if no Claude-User group overrides it. Because this fetcher genuinely honors robots.txt, an over-broad wildcard rule really does stop Claude from reading your pages on request - an outcome most other vendors' fetchers would have ignored.
How is Claude-User different from Claude-Web?
Claude-Web is a retired token. Claude-User is the current identifier for on-demand fetches made when a Claude user requests web access, and it is the one your robots.txt should name.
Should I allow Claude-User if I block ClaudeBot?
For most sites, yes. Blocking ClaudeBot keeps you out of training, while allowing Claude-User means a person who asks Claude about your page gets an answer based on what the page actually says.
Can Claude-User read your page right now?
Test any URL and see exactly what AI crawlers receive.