Meta-ExternalFetcher
Meta's user-triggered fetcher for Meta AI's agentic features. Meta states it may bypass robots.txt.
Reference reviewed by Sona
| Operator | Meta |
|---|---|
| Powers | Meta AI user-triggered and agentic link fetches |
| Purpose | User-triggered fetch |
| User-agent token | meta-externalfetcher |
| Respects robots.txt | Mostly |
Meta-ExternalFetcher fetches individual links at a user's request - the on-demand counterpart to Meta's bulk training crawl. Meta ties it explicitly to agentic AI capabilities, so it fires when Meta AI acts on a task rather than only when someone pastes a URL.
Meta's documentation states it may bypass robots.txt rules. That phrasing is softer than Perplexity's "generally ignores" but has the same practical implication: a Disallow for this token is a preference, not a control. If the block has to hold, it belongs in your WAF.
Meta AI's distribution is what gives this token weight. It is reachable from inside Facebook, Instagram, WhatsApp, and meta.ai, so the population that can trigger a fetch of your page is enormous, and fetches arrive in unpredictable bursts tied to conversations rather than to any crawl schedule.
Meta publishes no IP range feed or reverse-DNS convention for any of its crawlers, which is a real gap for this token specifically. It is the Meta bot most likely to reach content you asked it not to, and also the one you cannot authenticate - so edge rules end up keying on the user-agent string and, at best, on Meta's autonomous system.
How Meta-ExternalFetcher behaves
- Documented as possibly bypassing robots.txt, unlike Meta's training crawler.
- Covers agentic Meta AI activity, not only URLs a user pastes, so a single task can produce several fetches.
- Triggerable from Facebook, Instagram, WhatsApp, and meta.ai - a very large potential origin population.
- Arrives in conversational bursts with no crawl cadence to plan around.
Full user-agent string
meta-externalfetcher/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
Allow Meta-ExternalFetcher
Users and agents in Meta AI get answers grounded in your live page across Facebook, Instagram, WhatsApp, and meta.ai.
User-agent: meta-externalfetcher Allow: /
Block Meta-ExternalFetcher
You want no agentic or user-triggered Meta traffic - but enforce it at the WAF, because Meta documents that this token may bypass robots.txt.
User-agent: meta-externalfetcher Disallow: /
How to verify Meta-ExternalFetcher
No published verification method
Meta publishes neither an IP feed nor a reverse-DNS convention, so genuine fetches cannot be distinguished from impersonation. That gap is sharpest here: this is the Meta token documented as possibly bypassing robots.txt, so it is the one you are most likely to want to block precisely - and the one you can least confirm. An AS32934 routing lookup is the usual fallback, with the caveat that it is inferred rather than published.
Check an IP against this botCommonly confused with Meta-ExternalFetcher
The compliance split inside Meta: the training crawler respects robots.txt, this fetcher may bypass it. Rules for one say nothing about the other.
Same role at OpenAI, same weak robots.txt guarantee. Both need edge enforcement if a block must hold.
meta-webindexer builds Meta AI's search index ahead of time; this token fetches live during a task.
Meta-ExternalFetcher FAQs
Does Meta-ExternalFetcher respect robots.txt?
Meta's documentation states it may bypass robots.txt rules, on the basis that fetches are made at a user's request. Treat a Disallow as advisory and enforce at the server, CDN, or WAF if the block needs to hold.
Why am I seeing several fetches from one interaction?
Meta ties this token to agentic AI capabilities, so a single user task can involve fetching multiple links rather than one pasted URL. Bursts of a few requests are normal.
Can I verify Meta-ExternalFetcher traffic?
Not against a first-party source - Meta publishes no IP feed or reverse-DNS convention for its crawlers. Checking the address against Meta's AS32934 is the common workaround, but it is inferred rather than published verification.
Does blocking this affect link previews on Facebook or WhatsApp?
No. Share previews come from facebookexternalhit, a separate token. Blocking meta-externalfetcher stops Meta AI fetches without breaking how your links unfurl in posts and messages.
Can Meta-ExternalFetcher read your page right now?
Test any URL and see exactly what AI crawlers receive.