Meta-ExternalAgent
Meta's training crawler for Llama and Meta AI. Respects robots.txt, unlike two of its siblings.
Reference reviewed by Sona
| Operator | Meta |
|---|---|
| Powers | Meta AI and Llama model training |
| Purpose | Model training |
| User-agent token | meta-externalagent |
| Respects robots.txt | Yes |
Meta-ExternalAgent is the crawler Meta uses to collect web data for, in Meta's words, "training foundation AI models or improving products." It feeds the Llama model family and the Meta AI assistant surfaced across Facebook, Instagram, WhatsApp, and meta.ai.
Meta documents it as respecting robots.txt, and that is worth stating precisely because two of its siblings are documented as not. Meta-ExternalFetcher may bypass robots.txt, and facebookexternalhit may bypass it for security and integrity checks. Within Meta's crawler family, this training token is the compliant one - the opposite of the pattern at OpenAI and Perplexity, where the training crawler complies and the fetcher does not.
Note the token casing and form. Meta's tokens are lowercase and hyphenated - meta-externalagent, not Meta-ExternalAgent - and while robots.txt user-agent matching is case-insensitive, hand-written rules that guess at the shape of the name are a common source of silent no-ops. Copy the token exactly.
Meta now runs a four-token crawler family: meta-externalagent for training, meta-webindexer for the Meta AI search index, meta-externalfetcher for user-triggered link fetches, and meta-externalads for advertising and business products. Most robots.txt files written before this expansion name only one of them, so a rule set that was complete a year ago probably is not now.
How Meta-ExternalAgent behaves
- Documented as respecting robots.txt - the compliant member of Meta's crawler family.
- Uses a lowercase, hyphenated token; copy it exactly rather than guessing the casing.
- One of four Meta tokens, each with its own robots.txt group and its own compliance posture.
- No published IP feed or rDNS convention, so requests cannot be verified against a first-party source.
Full user-agent string
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/web-crawlers)
Allow Meta-ExternalAgent
Your content can inform Llama and Meta AI, which reach one of the largest user bases of any AI assistant.
User-agent: meta-externalagent Allow: /
Block Meta-ExternalAgent
Keep your content out of Meta's training pipeline - and unlike its fetcher siblings, a robots.txt Disallow here is documented to be honored.
User-agent: meta-externalagent Disallow: /
How to verify Meta-ExternalAgent
No published verification method
Meta publishes no IP range feed and no reverse-DNS convention for its crawler family, so there is no first-party way to confirm a request is genuinely Meta's. The usual fallback is to check whether the source address belongs to Meta's autonomous system (AS32934) using a routing lookup - reasonable evidence, but ASN data is not a crawler identity feed and should not be treated as one. Of the major AI operators, Meta is the notable one still not publishing verification data.
Check an IP against this botCommonly confused with Meta-ExternalAgent
The compliance split inside Meta: this training crawler respects robots.txt, while meta-externalfetcher may bypass it. Blocking one tells you nothing about the other.
meta-webindexer builds the Meta AI search index for citations. Blocking training here does not affect whether Meta AI can cite you.
FacebookBot is the older training token and no longer appears on Meta's crawler documentation page. meta-externalagent superseded it.
Meta-ExternalAgent FAQs
Does Meta-ExternalAgent respect robots.txt?
Yes, Meta documents this token as respecting robots.txt rules. That is specifically worth noting because meta-externalfetcher may bypass robots.txt and facebookexternalhit may bypass it for integrity checks - the compliance posture is not uniform across Meta's crawlers.
How many Meta crawlers do I need rules for?
Four are currently documented: meta-externalagent for training, meta-webindexer for the Meta AI search index, meta-externalfetcher for user-triggered fetches, and meta-externalads for advertising products. Older robots.txt files usually name only one.
Can I verify Meta crawler traffic?
Not from a first-party source. Meta publishes no IP range feed or reverse-DNS convention. Checking whether the address belongs to Meta's AS32934 is the common workaround, but that is a routing lookup rather than a crawler identity feed.
Does blocking this stop Meta AI from citing my site?
No. Citation eligibility comes from meta-webindexer, which builds the Meta AI search index. This token governs training data collection only.
Can Meta-ExternalAgent read your page right now?
Test any URL and see exactly what AI crawlers receive.