ExaBot
Exa's crawler - builds the semantic search index that many AI agents and applications query for web results.
Reference reviewed by Sona
| Operator | Exa |
|---|---|
| Powers | Exa's AI search API used by agents and AI apps |
| Purpose | Search & answers |
| User-agent token | ExaBot |
| Respects robots.txt | Yes |
ExaBot indexes the web for Exa (formerly Metaphor), a search API designed for AI: agents, assistants, and RAG applications call Exa to find and retrieve pages. Being in Exa's index makes you discoverable to a whole ecosystem of AI apps at once.
Not to be confused with the retired French search crawler 'Exabot' (Exalead). Exa's crawler respects robots.txt rules for its token.
Full user-agent string
Mozilla/5.0 (compatible; ExaBot/1.0; +https://exa.ai)
Allow ExaBot
One allow rule makes your content discoverable to the many AI agents and apps that search the web through Exa's API.
User-agent: ExaBot Allow: /
Block ExaBot
You don't want programmatic AI applications discovering and retrieving your pages via a search API.
User-agent: ExaBot Disallow: /
Can ExaBot read your page right now?
Test any URL and see exactly what AI crawlers receive.