Anthropic · Model training
ClaudeBot: Anthropic's training crawler, and how to block it
ClaudeBot is Anthropic's web crawler for model training: it collects public pages that may be used to train future Claude models. It honors robots.txt, so a two-line rule opts your site out. Blocking it does not stop Claude from finding or citing you in search answers; that is Claude-SearchBot's and Claude-User's job, and each has its own robots.txt token.
- robots.txt token
ClaudeBot- User agent
- Not published by the operator
- Operator
- Anthropic
- Type
- Model training
- What it feeds
- Web content that may be used to train Anthropic's Claude models, to improve their usefulness and safety.
- Follows robots.txt
- Yes. Anthropic says its bots honor robots.txt directives, and ClaudeBot also supports the non-standard Crawl-delay line.
- How to verify it
- Anthropic publishes the IP ranges its bots use (list)
Should you block ClaudeBot?
Blocking ClaudeBot tells Anthropic to leave your future pages out of model training. It has no effect on Claude's search features, which use Claude-SearchBot (indexing) and Claude-User (live fetches for a person's question). If you want to be cited by Claude but not trained on, block ClaudeBot and allow the other two.
User-agent: ClaudeBot
Disallow: /
User-agent: ClaudeBot
Allow: /
Add these lines to /robots.txt at your site root. A group naming ClaudeBot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is ClaudeBot allowed on your site?
ClaudeBot, Claude-SearchBot and Claude-User
Anthropic runs three crawlers, and they answer different questions:
| Token | What it does | Block it if… |
|---|---|---|
| ClaudeBot | Collects pages that may train future Claude models | You don't want your content used for training |
| Claude-SearchBot | Indexes pages to improve Claude's search results | You don't want Claude's search to surface you (rarely what you want) |
| Claude-User | Fetches a page when a person asks Claude about it | You don't want Claude to read pages people point it at |
Most sites that block "Claude" mean the first row and accidentally hit the other two with a wildcard. A rule such as User-agent: * with Disallow: / blocks all three, and a CDN "block AI bots" toggle often does the same.
What blocking ClaudeBot does, and what it doesn't
A Disallow for ClaudeBot is a signal that your future pages should be left out of Anthropic's training data. It is not retroactive, and it does not change how Claude answers questions today: when Claude searches the web it relies on Claude-SearchBot's index and on Claude-User fetches. That split is why "block training, allow search" is a coherent policy for a business that wants to be recommended.
If you run a publisher or a site whose value is the text itself, blocking ClaudeBot is a reasonable default. If you sell a product and want models to know what it is and who it is for, there is an argument for allowing it: what a model learns in training is what it can say about you before it searches.
How to block only ClaudeBot
Put this in /robots.txt at the root of each host:
User-agent: ClaudeBot
Disallow: /
A group that names ClaudeBot replaces your User-agent: * rules for it, so it no longer inherits paths you blocked for everyone. That is fine for a full block. For a partial block, repeat the shared rules inside the ClaudeBot group.
Checking that a visit is really ClaudeBot
Anyone can put "ClaudeBot" in a user agent. Anthropic publishes the IP ranges its crawlers use, so compare the request's source address with the bots.json list. The list covers Anthropic's bots together, so it tells you a request is from Anthropic, not which of the three bots sent it.
The live check above sends a request with ClaudeBot's token in the user agent and compares the answer with what a browser gets. It catches the common problem of a firewall or CDN challenging the bot even though robots.txt allows it. It can't reproduce Anthropic's real IP addresses, so rules keyed on IP may still behave differently.
Questions
- Does blocking ClaudeBot remove my site from Claude's answers?
- No. ClaudeBot is only for training. Claude finds pages to cite through Claude-SearchBot and fetches pages a person asks about through Claude-User. Leave those two allowed and Claude can still cite you.
- What is the ClaudeBot user agent string?
- Anthropic's help article names the robots.txt token, ClaudeBot, but does not publish a full user-agent string. Match on the token: any request whose user agent contains ClaudeBot claims to be it, and you can confirm it against Anthropic's published IP ranges.
- Can I slow ClaudeBot down instead of blocking it?
- Yes. Anthropic supports the Crawl-delay extension for its bots, for example "User-agent: ClaudeBot" followed by "Crawl-delay: 1". Google ignores Crawl-delay, so it only affects crawlers that honor it.
- Is it enough to block ClaudeBot by IP address?
- Anthropic warns that IP blocking may not opt you out reliably, because it also stops the crawler reading your robots.txt. Use a robots.txt rule as the opt-out, and add a firewall rule only if you also want to refuse the traffic.
- Do I need a rule on every subdomain?
- Yes. robots.txt applies per host, so blog.example.com and shop.example.com each need their own /robots.txt with the ClaudeBot rule.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.