AI crawler directory
Every major AI crawler, checked against its operator's own documentation: the robots.txt token, the user agent, what it feeds and how to verify a visit is real. The rule of thumb: allow search and user-triggered crawlers if you want to be cited in AI answers; training crawlers are your call.
Check which of these your robots.txt allows
AI search crawlers
These build the indexes AI answers cite. Block one and that engine can't recommend your pages.
- OAI-SearchBotOpenAIChatGPT's search features, where it is used to surface and link websites in answers.
- Claude-SearchBotAnthropicClaude's search results, by analyzing pages for the relevance and accuracy of search responses.
- PerplexityBotPerplexityPerplexity's search results, where it surfaces and links websites; Perplexity says it is not used for foundation-model training.
- GooglebotGoogleGoogle Search (including Discover and all Google Search features), plus other Google products such as Google Images, Video mode and Google News.
- BingbotMicrosoftThe Bing search index: Bing calls Bingbot its "standard crawler" that "handles most of our crawling needs each day".
- ApplebotAppleSearch across Apple's ecosystem (Spotlight, Siri, Safari), and possibly training of Apple foundation models and context for AI-generated answers in Apple products.
- Amzn-SearchBotAmazonSearch experiences in Amazon products and services, making permitted content eligible to appear in places such as Alexa.
- Meta-WebIndexerMetaMeta AI search: Meta says the crawler improves Meta AI's search result quality and helps it cite and link to your content in its responses.
- PetalBotHuaweiPetal Search, plus content recommendations in Huawei Assistant and AI Search, all powered by Petal Search.
- YouBotYou.comThe You.com search engine, providing comprehensive, real-time search results for You.com users.
User-triggered fetch crawlers
These fetch a page because a person asked an assistant about it. Blocking one breaks that answer.
- ChatGPT-UserOpenAILive page visits for user-initiated actions in ChatGPT and Custom GPTs, including GPT Actions; it does not crawl the web automatically.
- Claude-UserAnthropicLive page fetches when individuals ask Claude questions, supporting user-initiated requests.
- MistralAI-UserMistral AILive page visits when a user asks Mistral's assistant a question, so it can answer and cite the page; it does not crawl automatically or collect AI training data.
- DuckAssistBotDuckDuckGoReal-time page fetches for DuckDuckGo's AI-assisted answers, which cite their sources; the data is not used to train AI models.
Model training crawlers
These collect text to train future models. Blocking them doesn't remove you from AI search.
- GPTBotOpenAIContent that may be used to train OpenAI's generative AI foundation models and make them more useful and safe.
- ClaudeBotAnthropicWeb content that may be used to train Anthropic's Claude models, to improve their usefulness and safety.
- Google-ExtendedGoogleTraining of future Gemini models (Gemini Apps, Vertex AI API for Gemini) and grounding in Gemini Apps and Grounding with Google Search on Vertex AI.
- Applebot-ExtendedAppleDecides whether Applebot-crawled content may train Apple's general-purpose foundation models powering generative AI in Apple Intelligence, Services and Developer Tools.
- Meta-ExternalAgentMetaMeta says it crawls for use cases such as training foundation AI models or improving products by indexing content directly.
- AmazonbotAmazonImprovement of Amazon's products and services, including more accurate information for customers, and possibly training of Amazon AI models.
- CCBotCommon CrawlCommon Crawl's open repository of web crawl data, provided free for research and analysis and widely used as a source for AI training datasets.
- BytespiderByteDanceNot documented by ByteDance; third-party trackers report that it gathers data for ByteDance's AI models, which we could not confirm from a primary source.
Other crawling crawlers
Data and link-preview crawlers that are neither AI search nor model training.