How do I check if my robots.txt blocks GPTBot or ClaudeBot?

Enter your site above. The table shows, for each crawler, whether your homepage (and any path you type) is allowed, and which User-agent group the decision came from. Each crawler name links to its page in the AI crawler directory.

Does blocking AI crawlers in robots.txt remove me from ChatGPT or Claude answers?

Only if you block their search or user-triggered crawlers. Blocking GPTBot or ClaudeBot (training) leaves ChatGPT search (OAI-SearchBot) and Claude search (Claude-SearchBot, Claude-User) untouched. The "Allow AI search, block training" preset does exactly that split.

Why does the checker say a crawler is allowed when my firewall blocks it?

robots.txt is only a request. A CDN or firewall can still refuse or challenge a crawler that robots.txt allows. The live check on each crawler's page fetches your page as that crawler and compares the response with a browser's.

What happens if my robots.txt returns an error?

A missing file (a 4xx other than 429) means no restrictions. A server error or 429 makes crawlers back off: Google treats it as a full disallow at first and keeps using its last good copy for up to 30 days.

Is there a size limit for robots.txt?

Google reads the first 500 KiB and ignores the rest. Long files are usually generated lists that can be collapsed into a few wildcard rules.

Free tool · no sign-up

robots.txt checker and generator for AI crawlers

Enter a URL and this tool reads the site's robots.txt the way crawlers do (RFC 9309: the most specific user-agent group, the longest matching path, Allow wins ties). It tests every crawler in the AI crawler directory against your homepage and any path you type, lints the file for mistakes, and generates a corrected robots.txt from a preset.

robots.txt checker

Test your robots.txt against every AI crawler

robots.txt generator

Allow AI search, block trainingStay citable in ChatGPT, Claude, Perplexity, Gemini and Copilot answers while opting out of model training.Allow everythingEvery crawler may read every public page. The default when there is no robots.txt.Block all AI crawlersKeep Google and Bing search, block every AI crawler. AI answers will rarely cite or recommend the site.

Sitemap URLPrivate paths (one per line)

robots.txt

# robots.txt generated with https://www.getpicked.app/tools/robots-txt-checker
# Preset: Allow AI search, block training

User-agent: *
Allow: /

# Model training crawlers
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: CCBot
User-agent: Bytespider
Disallow: /

How the checker reads your file

Crawlers don't read robots.txt top to bottom. Each crawler looks for the group whose User-agent line names it most specifically, and only falls back to User-agent: * when no group names it. Inside that group the longest matching Allow or Disallow path wins, and Allow wins a tie. This is the behavior RFC 9309 standardizes and Google documents, and it is what the table above applies.

Two consequences catch people out:

A named group replaces *. If you write User-agent: GPTBot with one Disallow: /private/, GPTBot ignores everything in your * group, including Disallow: /checkout/. Repeat shared rules in every named group.
Names match by prefix of the token. A group for Applebot would also cover Applebot-Extended if there were no more specific group, which is why the checker reports the exact group it used.

What the lint warnings mean

Rules before any User-agent line are ignored by every crawler.
Paths that don't start with / are a common typo (Disallow: admin) that matches nothing.
Unknown directives such as Noindex: are not part of the standard. Google lists the fields it supports and noindex is not one of them; use a meta robots tag or X-Robots-Tag header instead.
Crawl-delay is ignored by Google. Some crawlers say they honor it (Anthropic's bots, CCBot, YouBot); others say they don't (Applebot, Amazonbot).
An HTML response at /robots.txt means your 404 page or app shell answered. Crawlers read that as "no rules".

Choosing a preset

Preset	Search and user-triggered crawlers	Training crawlers	Good for
Allow AI search, block training	Allowed	Blocked	Most businesses that want to be cited but not trained on
Allow everything	Allowed	Allowed	Sites that want maximum reach in AI models and answers
Block all AI crawlers	Blocked (Google and Bing stay allowed)	Blocked	Publishers who license content and accept losing AI citations

The generator writes your private paths into the * group and blocks crawlers in their own groups. Read the AI crawler directory before blocking anything you're unsure about: several crawlers are user-triggered fetchers whose operators say robots.txt may not apply, and blocking those takes a firewall rule.

Questions

How do I check if my robots.txt blocks GPTBot or ClaudeBot?: Enter your site above. The table shows, for each crawler, whether your homepage (and any path you type) is allowed, and which User-agent group the decision came from. Each crawler name links to its page in the AI crawler directory.
Does blocking AI crawlers in robots.txt remove me from ChatGPT or Claude answers?: Only if you block their search or user-triggered crawlers. Blocking GPTBot or ClaudeBot (training) leaves ChatGPT search (OAI-SearchBot) and Claude search (Claude-SearchBot, Claude-User) untouched. The "Allow AI search, block training" preset does exactly that split.
Why does the checker say a crawler is allowed when my firewall blocks it?: robots.txt is only a request. A CDN or firewall can still refuse or challenge a crawler that robots.txt allows. The live check on each crawler's page fetches your page as that crawler and compares the response with a browser's.
What happens if my robots.txt returns an error?: A missing file (a 4xx other than 429) means no restrictions. A server error or 429 makes crawlers back off: Google treats it as a full disallow at first and keeps using its last good copy for up to 30 days.
Is there a size limit for robots.txt?: Google reads the first 500 KiB and ignores the rest. Long files are usually generated lists that can be collapsed into a few wildcard rules.