Anthropic · AI search
Claude-SearchBot: Anthropic's search indexer, and why to allow it
Claude-SearchBot is Anthropic's search crawler: it navigates the web to improve the quality of Claude's search results, analyzing content for relevance and accuracy. Anthropic says its bots honor robots.txt. Blocking it may reduce how visible and accurate your site is in Claude's search answers, and it is separate from ClaudeBot, which handles training.
- robots.txt token
Claude-SearchBot- User agent
- Not published by the operator
- Operator
- Anthropic
- Type
- AI search
- What it feeds
- Claude's search results, by analyzing pages for the relevance and accuracy of search responses.
- Follows robots.txt
- Yes. Covered by Anthropic's general statement that its bots honor robots.txt directives; the help article gives no wording specific to Claude-SearchBot.
- How to verify it
- Anthropic publishes the IP ranges its bots use (list)
Should you block Claude-SearchBot?
Blocking Claude-SearchBot stops Anthropic indexing your pages for search, which may make your site less visible and less accurately described in Claude's search results. If you want Claude to recommend you, keep it allowed; to opt out of training, block ClaudeBot instead.
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
Add these lines to /robots.txt at your site root. A group naming Claude-SearchBot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is Claude-SearchBot allowed on your site?
Claude-SearchBot, ClaudeBot and Claude-User
Anthropic documents three crawlers in one help article, each with its own robots.txt token:
| Token | What it does | Block it if… |
|---|---|---|
| ClaudeBot | Collects pages that may train future Claude models | You don't want your content used for training |
| Claude-SearchBot | Indexes pages to improve Claude's search results | You don't want Claude's search to surface you |
| Claude-User | Fetches a page when a person asks Claude about it | You don't want Claude reading pages people point it at |
Claude-SearchBot is the middle row: it is not training and not a one-off fetch for a single person. It builds the picture of the web that Claude's search answers draw on.
What blocking Claude-SearchBot does, and what it doesn't
Anthropic says blocking it prevents Claude from indexing your content for search optimization, which may reduce your site's visibility and accuracy in Claude's search results. In practice, Claude is less likely to find you when someone asks a question you could answer, and more likely to describe you from older or second-hand sources.
It does not opt you out of training (ClaudeBot does that), and it does not stop Claude-User from fetching a page a person asks about. The common mistake is the reverse: a site meant to block training writes User-agent: * with Disallow: /, or turns on a CDN "block AI bots" setting, and loses search indexing too.
How to allow Claude-SearchBot and block training
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Once a bot has its own group, it ignores your User-agent: * group, so repeat any paths you want kept out of search. robots.txt applies per host, so blog.example.com and shop.example.com each need the rules in their own file. Anthropic's help article also documents support for the non-standard Crawl-delay line for its bots (its example uses ClaudeBot) if you want to slow crawling rather than block it.
Checking that a visit is really Claude-SearchBot
Anthropic doesn't publish a full user-agent string for Claude-SearchBot, so look for the token in the user agent and then compare the source IP with Anthropic's bots.json. The list is not split per bot: it tells you the request came from Anthropic, not which crawler sent it.
The live check above sends a request with Claude-SearchBot's token and compares the response with a browser's, which shows whether a firewall or CDN challenges the bot even though robots.txt allows it. It can't come from Anthropic's IPs, so IP-keyed rules may still differ. The robots.txt checker shows how each Anthropic token resolves against your file.
Questions
- Does blocking ClaudeBot also block Claude-SearchBot?
- No. They are separate robots.txt tokens. A ClaudeBot group only applies to ClaudeBot; Claude-SearchBot follows its own group, or your User-agent: * rules if it has none.
- What is Claude-SearchBot's user agent string?
- Anthropic's help article names the token but does not publish a full user-agent string. Match on Claude-SearchBot and confirm the IP against bots.json.
- What happens if I block Claude-SearchBot?
- Anthropic says it prevents Claude from indexing your content for search optimization, which may reduce your visibility and accuracy in Claude's search results.
- Can bots.json tell me which Anthropic bot visited?
- No. The list covers Anthropic's bots together, so it confirms a request came from Anthropic. Use the user-agent token to tell Claude-SearchBot from ClaudeBot or Claude-User.
- Is IP blocking a good way to stop Claude-SearchBot?
- Anthropic warns that blocking by IP may not reliably opt you out, because it also stops its crawler reading your robots.txt. Use a robots.txt rule if you want to opt out.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.