Meta · AI search
Meta-WebIndexer: Meta AI's search crawler, and why to allow it
Meta-WebIndexer is the crawler Meta uses to improve Meta AI's search results, and Meta says allowing it helps Meta AI cite and link to your content. It is separate from Meta-ExternalAgent, which crawls for AI training. If you want to show up in Meta AI answers, allow Meta-WebIndexer even if you block training.
- robots.txt token
Meta-WebIndexer- User agent
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)- Operator
- Meta
- Type
- AI search
- What it feeds
- Meta AI search: Meta says the crawler improves Meta AI's search result quality and helps it cite and link to your content in its responses.
- Follows robots.txt
- Yes. Meta tells site owners to add a robots.txt disallow for the relevant crawler and lists only Meta-ExternalFetcher and FacebookExternalHit as bots that may bypass it. Meta's page doesn't say in so many words that Meta-WebIndexer obeys robots.txt; that is inferred from this guidance.
- How to verify it
- Meta recommends allow-listing its crawlers by user agent or, more securely, by IP address, but its crawler page publishes no IP list, ASN or reverse-DNS method.
Should you block Meta-WebIndexer?
Blocking Meta-WebIndexer makes it less likely that Meta AI can cite and link to your pages in its answers. It does not stop training crawls, which run under Meta-ExternalAgent. Most sites that want AI visibility should allow it and decide about Meta-ExternalAgent separately.
User-agent: Meta-WebIndexer
Disallow: /
User-agent: Meta-WebIndexer
Allow: /
Add these lines to /robots.txt at your site root. A group naming Meta-WebIndexer replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is Meta-WebIndexer allowed on your site?
Meta's crawlers and what each one is for
Meta's Web Crawlers page lists five user agents. Only some of them are about AI, and two say they may ignore robots.txt:
| Token | What Meta says it does | Follows robots.txt? |
|---|---|---|
| Meta-WebIndexer | Improves Meta AI search results; helps Meta AI cite and link to your content | Yes, per Meta's general guidance |
| Meta-ExternalAgent | Use cases such as training foundation AI models or improving products by indexing content directly | Yes (Meta's own blocking example uses it) |
| Meta-ExternalAds | Improving advertising and other business products | Yes (also in Meta's example) |
| Meta-ExternalFetcher | Fetches individual links at a user's request, including for agentic AI tasks | May bypass robots.txt, because a user asked for the fetch |
| FacebookExternalHit | Builds link previews (title, description, thumbnail) when a link is shared on Facebook, Instagram or Messenger | May bypass robots.txt for security or integrity checks |
Meta-WebIndexer is the one that matters for being recommended by Meta AI. If you are unsure about FacebookBot, note that Meta's current page doesn't list it at all.
What blocking it does, and what it doesn't
A disallow for Meta-WebIndexer is a request that Meta stop crawling your pages for Meta AI search. The practical consequence is that Meta AI has less of your content to cite and link to. It doesn't opt you out of training (that is Meta-ExternalAgent), it doesn't stop link previews (FacebookExternalHit), and it won't stop Meta-ExternalFetcher when a person asks Meta AI to open your page.
How to allow or block it
Meta's example uses the lowercase token, so this page does too. To allow Meta AI search while blocking training:
User-agent: meta-webindexer
Allow: /
User-agent: meta-externalagent
Disallow: /
To block it, change its Allow: / to Disallow: /. Meta says crawlers may cache robots.txt for up to 24 hours, so give a change a day before judging it. robots.txt is read per host, so each subdomain needs its own file. You can check what your current file allows with the robots.txt checker.
Verifying a visit
Meta's page gives the strings meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers) and the shorter meta-webindexer/1.1. The link inside the first one is relative as Meta's page renders it; real traffic probably carries an absolute developers.facebook.com URL, but Meta doesn't say. Meta suggests allow-listing by IP address as the more secure option, yet it publishes no IP ranges, ASN or reverse-DNS hostname to allow-list against. Treat the user agent as a claim you can't independently confirm. Questions go to webmasters@meta.com, the contact on Meta's page.
Questions
- Does blocking Meta-ExternalAgent also block Meta-WebIndexer?
- No. They are separate robots.txt tokens. A group for meta-externalagent applies only to that crawler, so Meta-WebIndexer keeps following your other rules unless you name it or block everyone with a wildcard.
- Why does the user agent contain a relative link?
- Meta's page renders the string as "meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)". In live traffic the link is probably the full developers.facebook.com address, but Meta's docs don't confirm that, so match on the meta-webindexer token rather than the whole string.
- How do I confirm a request really came from Meta-WebIndexer?
- Meta doesn't give you a way to. It recommends allow-listing by IP address but publishes no IP list or reverse-DNS method on its crawler page, so the user agent is the only documented signal.
- How long does a robots.txt change take to apply?
- Meta says to allow up to 24 hours, because its crawlers may cache robots.txt for that long.
- Is Meta-WebIndexer used to train Meta's models?
- Meta describes it only as improving Meta AI search results and citations. Training is listed under Meta-ExternalAgent, which says it crawls for use cases such as training foundation AI models.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.