cohere-ai
Widely miscategorised as a training crawler - it actually retrieves pages for user-initiated prompts.
Reference reviewed by Sona
The operator publishes little or no documentation for this token. Details here come from community crawler registries and observed behavior, not vendor confirmation.
| Operator | Cohere |
|---|---|
| Powers | Live retrieval for Cohere-powered prompts |
| Purpose | User-triggered fetch |
| User-agent token | cohere-ai |
| Respects robots.txt | Mostly |
The cohere-ai token is routinely listed as a training crawler in published robots.txt templates, and that appears to be wrong. The community ai.robots.txt registry records its function as retrieving data for user-initiated LLM prompts - a live retrieval fetcher, not a bulk training collector. Cohere's separate cohere-training-data-crawler token is the one that gathers training data.
The mislabelling has a real cost. Publishers who add cohere-ai to a training block list, then stop, have blocked live retrieval while leaving the actual training crawler untouched - precisely inverting the intent. If your goal is to stay out of Cohere's training corpus, cohere-training-data-crawler is the token that matters.
Cohere publishes no dedicated crawler documentation page for either token, which is why this entry leans on community observation rather than vendor statements. Compliance is reported as standard but not confirmed in writing, and there is no published IP feed or verification method.
Cohere's models are aimed at enterprise search, retrieval, and generation deployments rather than a consumer chat product. So the allow-case here is less about brand visibility to the public and more about whether business tools built on Cohere can read your documentation when a user asks.
How cohere-ai behaves
- Community-documented as serving user-initiated prompts rather than bulk training collection.
- Frequently misfiled as a training crawler in published robots.txt templates.
- No vendor crawler documentation, no IP feed, and no verification method.
- Paired with cohere-training-data-crawler, which is the actual training token.
Allow cohere-ai
Enterprise tools built on Cohere can ground answers in your live pages when their users ask about you.
User-agent: cohere-ai Allow: /
Block cohere-ai
You want no programmatic retrieval by Cohere-powered applications - but pair it with cohere-training-data-crawler if training is your actual concern.
User-agent: cohere-ai Disallow: /
How to verify cohere-ai
No published verification method
Cohere publishes no IP range feed, no reverse-DNS convention, and no crawler documentation page, so cohere-ai traffic cannot be authenticated. Given how commonly the token appears in copy-pasted block lists, it is also a plausible identity for a scraper to borrow. Edge rules here have to match the user-agent string on trust.
Check an IP against this botCommonly confused with cohere-ai
This is the pairing to get right. cohere-training-data-crawler collects training data; cohere-ai serves live prompts. Blocking only cohere-ai leaves the training crawl running.
A comparable user-triggered retrieval token from another vendor - and one that, unlike this one, is properly documented with a published IP feed.
cohere-ai FAQs
Is cohere-ai a training crawler?
Apparently not, despite how often it is listed as one. The ai.robots.txt registry records its function as retrieving data for user-initiated LLM prompts. Cohere's training collection runs under the separate cohere-training-data-crawler token.
I blocked cohere-ai to opt out of Cohere training. Did that work?
Probably not. You have likely blocked live retrieval while leaving the training crawler running. Add a rule for cohere-training-data-crawler, which is the token documented as collecting training data.
Why is the information here less definitive than for other bots?
Cohere publishes no crawler documentation page for either of its tokens. Everything known comes from community observation in registries such as ai.robots.txt, so treat compliance as reported rather than vendor-confirmed.
Can I verify cohere-ai traffic?
No. There is no published IP feed and no reverse-DNS convention. Since the token appears in many copy-pasted block lists, it is also an easy identity for an unrelated scraper to adopt.
Can cohere-ai read your page right now?
Test any URL and see exactly what AI crawlers receive.