Meta-WebIndexer
Meta's search-index crawler. This is the token that decides whether Meta AI can cite you - not the training one.
Reference reviewed by Sona
| Operator | Meta |
|---|---|
| Powers | Meta AI search results and citations |
| Purpose | Search & answers |
| User-agent token | meta-webindexer |
| Respects robots.txt | Yes |
Meta-WebIndexer crawls to build the index behind Meta AI's search experiences. Meta's description is that it "navigates the web to improve Meta AI search result quality for users" - an indexing crawler, not a training crawler, and the one that governs whether your pages can be cited and linked in Meta AI responses.
It is the newest addition to Meta's crawler family and the most commonly missing from existing robots.txt files. That gap tends to cut the wrong way: sites that added a broad Disallow years ago to keep out of Meta's training are now also excluded from Meta AI citations, without having chosen that.
Meta now runs four documented tokens, and knowing which does what is the whole game: meta-externalagent collects training data, meta-webindexer builds the search index, meta-externalfetcher performs user-triggered and agentic link fetches, and meta-externalads supports advertising products. Each reads its own robots.txt group, so allowing citations while refusing training is straightforward once you name the right tokens.
The distribution argument is hard to ignore. Meta AI is reachable from inside Facebook, Instagram, and WhatsApp, which puts it in front of a larger population than any standalone AI assistant. Citation eligibility there is a meaningful visibility channel even if the referral behavior is less mature than Google's.
How Meta-WebIndexer behaves
- The newest and most frequently omitted Meta token, so old robots.txt files often block it unintentionally.
- Indexes for citation rather than collecting training data.
- Documented without a robots.txt bypass exception, unlike meta-externalfetcher and facebookexternalhit.
- No published IP feed or reverse-DNS convention.
Full user-agent string
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
Allow Meta-WebIndexer
Your pages become citable, linkable sources inside Meta AI across Facebook, Instagram, and WhatsApp - an enormous potential audience.
User-agent: meta-webindexer Allow: /
Block Meta-WebIndexer
You don't want your content surfaced in Meta AI's answers, accepting the loss of citation visibility across Meta's apps.
User-agent: meta-webindexer Disallow: /
How to verify Meta-WebIndexer
No published verification method
As with the rest of Meta's crawler family, there is no published IP range feed and no reverse-DNS convention, so requests cannot be authenticated against a first-party source. Meta is now the largest AI operator still not publishing verification data - OpenAI, Anthropic, Google, Microsoft, Perplexity, Amazon, Apple, and Mistral all do. An AS32934 routing lookup is the usual substitute, with the caveat that it is inferred.
Check an IP against this botCommonly confused with Meta-WebIndexer
The pairing that matters: meta-externalagent collects training data, this token decides whether Meta AI can cite you. Blocking both when you meant to block training only costs you visibility.
This indexes ahead of time; the fetcher retrieves live during a user task, and unlike this token it may bypass robots.txt.
OpenAI's equivalent citation crawler. If you are setting an AI-search visibility policy, these two usually belong on the same side of it.
Meta-WebIndexer FAQs
Which Meta token controls whether Meta AI can cite my site?
meta-webindexer. It builds the index behind Meta AI's search experiences. The training token, meta-externalagent, has no bearing on citation eligibility.
My robots.txt blocks Meta broadly. What did that cost me?
Probably your Meta AI citations. Rules written before this token existed usually fall through to a wildcard Disallow, which excludes you from Meta AI answers as well as from training - likely not what you intended.
How many Meta crawler tokens are there now?
Four documented: meta-externalagent for training, meta-webindexer for the search index, meta-externalfetcher for user-triggered and agentic fetches, and meta-externalads for advertising products.
Can I verify Meta-WebIndexer traffic?
Not from a first-party source. Meta publishes no IP feed or reverse-DNS convention for any of its crawlers, which makes it the largest AI operator without published verification data.
Can Meta-WebIndexer read your page right now?
Test any URL and see exactly what AI crawlers receive.