Meta-WebIndexer
Meta's search-index crawler. Controls whether your pages can surface and be cited in Meta AI's search answers.
Reference reviewed by Sona
| Operator | Meta |
|---|---|
| Powers | Meta AI search results and citations |
| Purpose | Search & answers |
| User-agent token | meta-webindexer |
| Respects robots.txt | Yes |
Meta-WebIndexer crawls the web purely to build the index behind Meta AI's search experiences. Meta documents that allowing it makes your content eligible to be cited and linked in Meta AI responses - it is an indexing crawler, not a training crawler.
It is the third leg of Meta's crawler family: meta-externalagent collects training data, meta-externalfetcher performs user-triggered link fetches, and meta-webindexer feeds the search index. Each has its own robots.txt token, so you can allow citations while opting out of training.
Full user-agent string
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)
Allow Meta-WebIndexer
Your pages become citable, linkable sources inside Meta AI's answers across Facebook, Instagram, and WhatsApp surfaces - a large referral audience.
User-agent: meta-webindexer Allow: /
Block Meta-WebIndexer
You don't want your content surfaced inside Meta AI's answer interfaces.
User-agent: meta-webindexer Disallow: /
Can Meta-WebIndexer read your page right now?
Test any URL and see exactly what AI crawlers receive.