Claude-User: Anthropic's on-demand fetcher, and how it uses robots.txt

Claude-User, ClaudeBot and Claude-SearchBot

Anthropic runs three crawlers, each with its own robots.txt token:

Token	What it does	Block it if…
ClaudeBot	Collects pages that may train future Claude models	You don't want your content used for training
Claude-SearchBot	Indexes pages to improve Claude's search results	You don't want Claude's search to surface you
Claude-User	Fetches a page when a person asks Claude about it	You don't want Claude reading pages people point it at

Claude-User is the only one that runs because a specific person asked something. It is the closest Anthropic equivalent to OpenAI's ChatGPT-User and Perplexity's user fetcher (covered on the PerplexityBot page), with one difference that matters below.

Robots.txt applies here, unlike some rivals

OpenAI says robots.txt "may not apply" to ChatGPT-User, and Perplexity says its user fetcher generally ignores it. Anthropic gives no such exemption: it says Claude-User lets site owners control which sites user-initiated requests can reach, and that its bots honor robots.txt. That is Anthropic's own description; we have not independently tested it.

The practical consequence is that your robots.txt has more effect on Claude-User than on similar bots. A site-wide User-agent: * with Disallow: /, written to keep training crawlers away, will also stop Claude reading your pricing page when a buyer asks about it.

What blocking Claude-User does, and what it doesn't

Anthropic says blocking it prevents Claude from retrieving your content in response to a user query, which may reduce your visibility for user-directed web search. It does not affect training (ClaudeBot) or Claude's search index (Claude-SearchBot).

How to allow Claude-User while blocking training

User-agent: ClaudeBot
Disallow: /

User-agent: Claude-User
Allow: /

A named group overrides your wildcard group for that bot, so repeat any private paths inside it. Rules apply per host, so each subdomain needs its own /robots.txt. Anthropic warns that blocking by IP may not reliably opt you out, because it also stops the crawler reading robots.txt.

Checking that a visit is really Claude-User

Anthropic doesn't publish a full user-agent string for Claude-User. Look for the token, then check the source address against bots.json. That list covers all of Anthropic's bots, so it confirms Anthropic, not which bot.

The live check above sends a request with the Claude-User token and compares it with a browser request, which catches CDN or firewall blocks that robots.txt can't explain. It doesn't come from Anthropic's IPs, so IP-keyed rules may still differ. The robots.txt checker shows whether a wildcard rule is catching Claude-User.