OpenAI · Model training
GPTBot: OpenAI's training crawler, and how to block it
GPTBot is OpenAI's web crawler for model training: it collects public pages that may be used to train OpenAI's generative AI foundation models. Disallowing it in robots.txt tells OpenAI your content should not be used for that training. It does not control whether you appear in ChatGPT search; OAI-SearchBot does, and the two settings are independent.
- robots.txt token
GPTBot- User agent
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot- Operator
- OpenAI
- Type
- Model training
- What it feeds
- Content that may be used to train OpenAI's generative AI foundation models and make them more useful and safe.
- Follows robots.txt
- Yes. OpenAI's docs say that disallowing GPTBot indicates a site's content should not be used in training generative AI foundation models.
- How to verify it
- OpenAI publishes the IP ranges GPTBot uses (list)
Should you block GPTBot?
Blocking GPTBot asks OpenAI to keep your content out of foundation-model training. Per OpenAI's docs it has no effect on ChatGPT search, which OAI-SearchBot controls. If you want ChatGPT to cite you but not train on you, block GPTBot and allow OAI-SearchBot.
User-agent: GPTBot
Disallow: /
User-agent: GPTBot
Allow: /
Add these lines to /robots.txt at your site root. A group naming GPTBot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is GPTBot allowed on your site?
GPTBot and OpenAI's other crawlers
OpenAI's crawler documentation now lives at developers.openai.com/api/docs/bots; the old platform.openai.com/docs/bots address redirects there. It lists four user agents:
| Token | What it does | Honors robots.txt |
|---|---|---|
| GPTBot | Crawls content that may train OpenAI's foundation models | Yes |
| OAI-SearchBot | Surfaces and links sites in ChatGPT's search features | Yes |
| ChatGPT-User | Visits a page when a ChatGPT or Custom GPT user's request needs it | "May not apply" |
| OAI-AdsBot | Checks landing pages submitted as ChatGPT ads | Only visits submitted ad pages |
OpenAI states that each robots.txt setting is independent. A common policy is to allow OAI-SearchBot so ChatGPT can surface you, and disallow GPTBot so the crawled content is not used for training.
What blocking GPTBot does, and what it doesn't
A Disallow for GPTBot tells OpenAI your content should not be used to train its generative AI foundation models. It does not stop ChatGPT from showing your site in search answers, and it does not stop ChatGPT-User from opening a page a user asks about.
One quirk matters if you allow both training and search: OpenAI says it may use the results from just one crawl for both use cases. So the requests you see from OpenAI may serve two purposes at once. Blocking GPTBot removes the training use without affecting search, and that is the only lever you have over training.
How to block only GPTBot
Add this to /robots.txt at the root of each host:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
The second group is optional, but it makes your intent explicit. A group that names GPTBot replaces your User-agent: * rules for it, so for a partial block, copy any shared Disallow lines into the GPTBot group too. A wildcard User-agent: * with Disallow: / blocks GPTBot and OAI-SearchBot alike.
Checking that a visit is really GPTBot
OpenAI's example string is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot, and OpenAI says the version number may change. When GPTBot fetches robots.txt it may add a robots.txt marker to that string. Because any client can copy the user agent, compare the source IP with OpenAI's published GPTBot ranges.
The live check above sends a request with GPTBot's token and compares the response with what a browser gets, which catches a CDN or firewall rule that blocks the bot even when robots.txt allows it. It can't reproduce OpenAI's IP addresses, so IP-based rules may still behave differently. You can also test your rules with the robots.txt checker.
Questions
- Does blocking GPTBot remove my site from ChatGPT search?
- No. OpenAI says each setting is independent: OAI-SearchBot decides whether you appear in ChatGPT search answers, and GPTBot only signals training use. Keep OAI-SearchBot allowed if you want to be cited.
- Why does the GPTBot version number in my logs differ from this page?
- OpenAI labels its GPTBot string an example and says the version number may change. Match on the GPTBot token rather than the full string, and confirm the IP against gptbot.json.
- Why do I see one OpenAI crawl when I allow both GPTBot and OAI-SearchBot?
- OpenAI says that if a site allows both bots, it may use the results of a single crawl for both purposes to avoid duplicate crawling. Blocking either one removes that use.
- Is the user agent different when GPTBot fetches robots.txt?
- It can be. OpenAI says it may add a robots.txt marker to the user-agent string when fetching robots.txt, so you can tell those requests apart in logs that don't record paths.
- Does blocking GPTBot remove content OpenAI already collected?
- OpenAI's crawler page describes the rule as a signal about training use and does not say it is retroactive. Treat it as applying to future crawls.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.