AI crawler checker
Free check of 20 AI crawlers, including OAI-SearchBot, GPTBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Bingbot, Googlebot, Meta-WebIndexer and Applebot, against your robots.txt and firewall, plus the indexes and shopping programs each AI agent reads.
Direct answer
Each AI assistant reads the web through its own crawler or a partner index: ChatGPT uses OAI-SearchBot plus partner search providers, Claude searches with Brave, Copilot uses Bing, Gemini and AI Overviews use Google, and Perplexity runs its own index. If robots.txt or a firewall blocks one of these crawlers, that assistant cannot read or cite your pages. This checker requests your pages as each crawler and reads your robots.txt rules for each one.
Agent coverage
We request your homepage and a product page as each AI crawler, read your robots.txt rules for each one, and check the indexing and shopping signals agents rely on. Account-level items (Search Console, Bing Webmaster, Merchant Center, Shopify channels) are listed for you to confirm.
Many sites block AI crawlers by accident: CDN defaults, security apps, and copy-pasted robots.txt rules often stop search crawlers such as OAI-SearchBot or Claude-SearchBot along with training crawlers. Blocking a search crawler removes you from that assistant's answers today; blocking a training crawler limits what its future models know about you.
| Signal | What to check | Why AI systems care |
|---|---|---|
| Search crawlers | Allow Googlebot, Bingbot, OAI-SearchBot, PerplexityBot, Claude-SearchBot, Meta-WebIndexer, Applebot, Amzn-SearchBot. | They decide whether assistants can cite you at all. |
| User-triggered fetchers | Do not block ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User in your firewall. | They fetch your page when a shopper asks about it directly. |
| Training crawlers | Decide deliberately on GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot. | They shape what models know without searching. |
| Three indexes | Be indexed in Google, Bing and Brave. | Together they feed nearly every assistant's live web results. |
Worked example
A store's robots.txt allows every crawler, but its CDN's AI-bot setting returns 403 to OAI-SearchBot and PerplexityBot. Google and Bing still index it, so the problem is invisible in Search Console, yet ChatGPT and Perplexity cannot read its product pages and recommend competitors instead.
robots.txt: allowed for every crawler.
Firewall: 403 for two AI search crawlers.
Fix: allow verified AI search crawlers in the CDN's bot settings.
No. OpenAI uses separate crawlers: GPTBot for training and OAI-SearchBot for ChatGPT search. Blocking GPTBot affects training only; blocking OAI-SearchBot removes you from ChatGPT search results.
Anthropic lists Brave Search as its web search provider, and Claude Code's web search uses Brave. Brave only crawls pages that Googlebot is allowed to crawl, so blocking Googlebot also hides you from Claude.
We test with each crawler's user agent from our servers. Some firewalls verify the real crawler by IP address, so they may block our test but allow the genuine bot. Treat a 'Check' result as a prompt to review your firewall's AI-bot settings.
No. AI answers vary by model, prompt, location, time, and browsing surface. GetPicked improves readiness, clarity, evidence, and measurement so products are easier for AI systems to understand and recommend.
GetPicked is built for independent Shopify brands that sell differentiated products and want a practical audit plus publish-ready fixes instead of another analytics dashboard. Beauty, skincare, supplements, fashion, pet, and home goods are the most common categories we audit.
Platform details change often. Checked on Oct 7, 2026.
GEO for Shopify: make product pages easier for AI to cite and recommend
A practical guide to generative engine optimization for Shopify brands: what AI answer engines need from product pages, and what to fix first.
AI search optimization for Shopify stores
How Shopify merchants can prepare product pages for ChatGPT, Perplexity, Gemini, Copilot, and AI shopping agents.
ChatGPT shopping optimization for Shopify products
Prepare Shopify product pages and feeds for ChatGPT-style product discovery and AI shopping recommendations.
AI Pickability Score for Shopify products
A 0-100 score that shows whether AI shopping agents can understand, trust, compare, and recommend Shopify products.
AI product visibility audit for Shopify brands
Audit whether AI systems can access, understand, cite, and recommend your Shopify product pages.