Meta · Model training
Meta-ExternalAgent: Meta's AI training crawler, and how to block it
Meta-ExternalAgent is the Meta crawler that, in Meta's words, crawls the web for use cases such as training foundation AI models or improving products by indexing content directly. Meta's own robots.txt example uses it, so a disallow opts your site out. Blocking it does not remove you from Meta AI's cited answers, which rely on Meta-WebIndexer.
- robots.txt token
Meta-ExternalAgent- User agent
meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)- Operator
- Meta
- Type
- Model training
- What it feeds
- Meta says it crawls for use cases such as training foundation AI models or improving products by indexing content directly.
- Follows robots.txt
- Yes. Meta's robots.txt example names meta-externalagent and tells site owners to add a disallow to block it. Changes can take up to 24 hours because Meta's crawlers may cache robots.txt that long.
- How to verify it
- Meta recommends allow-listing its crawlers by user agent or, more securely, by IP address, but its crawler page publishes no IP list, ASN or reverse-DNS method.
Should you block Meta-ExternalAgent?
Blocking Meta-ExternalAgent asks Meta to keep your future pages out of foundation-model training and out of the direct-indexing product uses Meta lists for it. Meta AI can still cite you through Meta-WebIndexer if you allow that crawler. Block both only if you want no Meta AI use of your content at all.
User-agent: Meta-ExternalAgent
Disallow: /
User-agent: Meta-ExternalAgent
Allow: /
Add these lines to /robots.txt at your site root. A group naming Meta-ExternalAgent replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is Meta-ExternalAgent allowed on your site?
Where Meta-ExternalAgent sits among Meta's crawlers
Meta's Web Crawlers page splits its traffic into five user agents. For an AI policy, these are the ones to tell apart:
| Token | Purpose per Meta | Controlled by robots.txt? |
|---|---|---|
| Meta-ExternalAgent | Use cases such as training foundation AI models or improving products by indexing content directly | Yes |
| Meta-WebIndexer | Improving Meta AI search; helps Meta AI cite and link to you | Yes, per Meta's general guidance |
| Meta-ExternalAds | Advertising and other business products | Yes |
| Meta-ExternalFetcher | Fetching a link at a user's request, including for agentic AI | May bypass robots.txt |
| FacebookExternalHit | Link previews when a URL is shared in Meta's apps | May bypass robots.txt for security or integrity checks |
The older FacebookBot token no longer appears on Meta's page.
What blocking it does, and what it doesn't
A disallow for Meta-ExternalAgent is the opt-out Meta documents for training and direct indexing. It applies going forward and to that token only. It leaves Meta-WebIndexer, link previews and user-requested fetches untouched, so a page can still be cited by Meta AI, shared with a preview card, or opened by Meta-ExternalFetcher when someone asks Meta AI about it. If what you want is "no training, but stay citable", block this crawler and allow Meta-WebIndexer.
How to block it
Meta's own example uses the lowercase token:
User-agent: meta-externalagent
Disallow: /
User-agent: meta-webindexer
Allow: /
Meta's example also shows a partial block (Allow: / plus Disallow: /private/) if you only want to keep some directories out. Meta says crawlers may cache robots.txt for up to 24 hours, so a change can take a day to apply. Each subdomain serves its own robots.txt and needs its own rule. The robots.txt checker shows how your file treats each Meta token.
Verifying a visit
The documented strings are meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers) and the short form meta-externalagent/1.1. That link is relative as Meta's page renders it; in your logs it is probably an absolute developers.facebook.com URL, but Meta's docs don't confirm the exact form, so match on the token. Meta suggests allow-listing by IP address but gives no IP list, ASN or reverse-DNS hostname, so there is no published way to prove a request came from Meta. Questions can go to webmasters@meta.com.
Questions
- Will blocking Meta-ExternalAgent stop Meta AI from citing my site?
- Not by itself. Meta says Meta-WebIndexer is the crawler that helps Meta AI cite and link to your content. Leave meta-webindexer allowed and only training and direct-indexing crawls are affected.
- Does blocking it stop Facebook link previews?
- No. Link previews come from FacebookExternalHit, which has its own token. Blocking Meta-ExternalAgent leaves shared-link cards working.
- Is Meta-ExternalAgent only for training?
- Meta lists two example uses: training foundation AI models, and improving products by indexing content directly. It says "such as", so the list isn't presented as complete, and a single disallow covers all of them.
- Can I block it by IP address?
- Meta doesn't publish its IP ranges on its crawler page, even though it calls IP allow-listing the more secure option. A robots.txt disallow is the documented way to opt out; a user-agent rule at your firewall or CDN is the fallback if you also want to refuse the requests.
- Does Meta-ExternalAgent cover Meta-ExternalFetcher too?
- No. Meta-ExternalFetcher is a separate crawler that fetches links a user asked for, and Meta says it may bypass robots.txt, so a meta-externalagent rule doesn't reach it.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.