YouBot
You.com's crawler. Unusually, one token covers both search indexing and LLM training - you cannot split them.
Reference reviewed by Sona
| Operator | You.com |
|---|---|
| Powers | You.com AI search answers and model training |
| Purpose | Search & answers |
| User-agent token | YouBot |
| Respects robots.txt | Yes |
YouBot builds the index behind You.com, one of the earliest AI-native search engines, which answers queries with generated summaries and linked citations. It honors robots.txt and supports Crawl-delay, so throttling is available as an alternative to blocking.
The distinguishing feature is an absence. The ai.robots.txt registry records YouBot as retrieving data for the search engine and for LLM training - both under one token. Nearly every other operator in this directory splits those roles, letting you stay citable while opting out of training. You.com does not, so allowing YouBot means accepting both and blocking it means giving up both.
That single-token design is the whole decision here. If your policy is the common "citable but not training data" posture, YouBot is the entry where that policy has no clean expression - you have to pick which half matters more.
There is a compliance footnote worth knowing. Although You.com documents robots.txt compliance, TollBit's 2026 measurement grouped YouBot with ChatGPT-User and Bytespider among agents observed reaching disallowed pages on a substantial share of the European sites that listed them. Documented compliance and observed compliance are not the same thing.
How YouBot behaves
- One token covers both search indexing and LLM training, with no way to separate them.
- Honors Crawl-delay, so rate limiting is a supported middle option.
- Documented as respecting robots.txt, though 2026 third-party measurement observed it on disallowed pages.
- No published IP feed or reverse-DNS convention, so requests cannot be verified.
Full user-agent string
Mozilla/5.0 (compatible; YouBot (+http://www.you.com))
Allow YouBot
Eligibility to be cited in You.com's AI answers - accepting that the same access also feeds LLM training.
User-agent: YouBot Allow: /
Block YouBot
The only way to opt out of You.com's training use, since one token covers both training and search indexing.
User-agent: YouBot Disallow: /
How to verify YouBot
No published verification method
You.com publishes no IP range feed or reverse-DNS convention, so YouBot traffic cannot be authenticated. That matters more here than for a purely documented-and-compliant crawler, because independent measurement has recorded YouBot reaching disallowed pages - and without verification you cannot tell whether such a hit was You.com ignoring your rules or someone else wearing the name.
Check an IP against this botCommonly confused with YouBot
OpenAI separates search indexing from training across two tokens; You.com combines both into YouBot, so the same policy cannot be expressed here.
Both index for AI-native search, but Exa serves an API consumed by other applications while You.com runs its own consumer answer engine.
YouBot FAQs
Can I be cited by You.com without feeding its training data?
No. Community documentation records YouBot as serving both the search engine and LLM training under a single token, so the two cannot be separated. Allowing it accepts both; blocking it forfeits both.
Does YouBot support Crawl-delay?
Yes. It honors robots.txt and supports Crawl-delay, so throttling the crawl rate is a genuine middle option rather than an all-or-nothing choice.
Does YouBot actually honor robots.txt?
It is documented as doing so, but TollBit's 2026 reporting grouped it with ChatGPT-User and Bytespider among agents observed reaching disallowed pages on a substantial share of European sites that had listed them. Verify against your own logs.
Is You.com worth being indexed by?
It is a smaller audience than ChatGPT or Perplexity, but it is citation-forward and AI-native. The real question is whether that visibility is worth the training exposure bundled with it.
Can YouBot read your page right now?
Test any URL and see exactly what AI crawlers receive.