GetPicked

AI search glossary

robots.txt: what it is and how to set it for AI crawlers

robots.txt is a plain text file at the root of a site (/robots.txt) that tells crawlers which URLs they may fetch, using user-agent groups with allow and disallow rules. It follows the Robots Exclusion Protocol, standardized as RFC 9309.

Also called: robots txt, robots exclusion protocol, REP, robots file

Why it matters for AI search

robots.txt is the first gate for AI visibility. Each AI company runs separate crawlers for training and for search, and blocking the search one (OAI-SearchBot for ChatGPT search, for example) can remove you from its answers. Many sites block AI bots wholesale with one copied rule and lose citations they wanted. Check each bot by name and decide on purpose.

How robots.txt works

The file must live at the top level of the host, for example https://example.com/robots.txt, as UTF-8 plain text. It is made of groups. Each group starts with one or more User-agent lines, followed by Allow and Disallow rules:

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Disallow: /cart/
Sitemap: https://example.com/sitemap.xml

RFC 9309 sets the matching rules. A crawler uses the group that names its product token, matched case-insensitively, and falls back to the * group if none does. Within a group, the most specific (longest) matching path wins, and a URL with no matching rule is allowed. A 4xx response for the file means crawlers may fetch anything; a 5xx means they must assume everything is disallowed.

Google says robots.txt is used mainly to avoid overloading a site with requests and to keep crawlers away from unimportant pages. It is not access control: the paths you list are public.

AI companies split their crawlers by job. OpenAI documents GPTBot for training, OAI-SearchBot for ChatGPT search, and ChatGPT-User for fetches a person triggers. Anthropic runs ClaudeBot and Claude-SearchBot; Perplexity runs PerplexityBot. Google uses Googlebot for Search and AI Overviews, and the Google-Extended token for training controls.

OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. A blanket Disallow: / for every AI bot therefore trades away citations, not only training use.

How to check and fix yours

  1. Open /robots.txt on your live domain and read every group, including rules added by plugins or your CDN.
  2. Run the AI crawler checker to see which named AI bots are allowed or blocked.
  3. Allow the search crawlers you want to be cited by, and decide separately on training crawlers.
  4. Do not use robots.txt to hide pages from results; use a robots meta tag with noindex.
  5. Reference your XML sitemap with a Sitemap: line.

Rules match on the bot's user agent token, and the full list of bots lives in the AI crawlers directory. For whether a page can be fetched and indexed at all, see crawlability.

Questions

Does robots.txt keep a page out of Google?
No. Google says robots.txt is not a mechanism for keeping a page out of Google. A blocked URL can still be indexed without a description if other pages link to it. Use a noindex rule or password protection instead, and leave the page crawlable so Google can see the noindex.
Should I block AI crawlers in robots.txt?
Decide per bot. Training crawlers such as GPTBot and search crawlers such as OAI-SearchBot do different jobs. Blocking training while allowing search keeps you eligible for ChatGPT search answers. Blocking everything removes you from AI answers that rely on those crawlers.
Do all bots obey robots.txt?
No. The RFC says the rules are not a form of access authorization, and Google notes that reputable crawlers obey it but others may not. OpenAI also says robots.txt rules may not apply to ChatGPT-User, because a person starts those fetches.
How fast do changes to robots.txt take effect?
RFC 9309 says crawlers should not use a cached copy for more than 24 hours unless the file is unreachable, so most changes are picked up within a day. Reindexing or dropping pages can take longer.

Related terms

Sources