Common Crawl · Model training
CCBot: Common Crawl's open-data crawler, and how to block it
CCBot is the Common Crawl Foundation's web crawler: it builds Common Crawl's open repository of web crawl data, which Common Crawl provides free for research and analysis. That corpus is widely used to build AI training datasets, which is why our directory classes CCBot under training. CCBot honors robots.txt, including Crawl-delay, but blocking it only affects future crawls.
- robots.txt token
CCBot- User agent
CCBot/2.0 (https://commoncrawl.org/faq/)- Operator
- Common Crawl
- Type
- Model training
- What it feeds
- Common Crawl's open repository of web crawl data, provided free for research and analysis and widely used as a source for AI training datasets.
- Follows robots.txt
- Yes. Common Crawl's FAQ says a Disallow for CCBot stops it crawling your site, and that CCBot obeys Crawl-delay, honors nofollow on links, uses Sitemaps listed in robots.txt and rechecks robots.txt periodically.
- How to verify it
- Common Crawl publishes CCBot's IP ranges; also check forward-confirmed reverse DNS (hostnames like 18-97-14-84.crawl.commoncrawl.org). IPv6 addresses don't have reverse DNS yet. (list)
Should you block CCBot?
Blocking CCBot keeps your pages out of future Common Crawl snapshots, so they are less likely to end up in research datasets and AI models trained on that data. CCBot powers no search engine, so you lose no direct search traffic. Snapshots already published stay public.
User-agent: CCBot
Disallow: /
User-agent: CCBot
Allow: /
Add these lines to /robots.txt at your site root. A group naming CCBot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is CCBot allowed on your site?
What CCBot is, and why it matters for AI
CCBot is run by the Common Crawl Foundation, which publishes an open archive of web crawl data rather than building AI products itself. On its CCBot page and FAQ, Common Crawl describes the data as free for research and analysis, and neither page talks about AI training. In practice its snapshots are one of the most common starting points for AI training datasets, so we file CCBot next to GPTBot, ClaudeBot and Google-Extended. That classification is ours.
Common Crawl doesn't run a search engine or answer engine. Unlike OAI-SearchBot or PerplexityBot, nothing a user searches for today reads from CCBot directly.
What blocking it does, and what it doesn't
Blocking CCBot is a forward-looking opt-out. New snapshots stop including your pages, so models trained on newer Common Crawl data are less likely to have seen them. It doesn't reach back into archives already released, and it can't stop a model builder who has already downloaded them. It also doesn't affect any AI company's own crawler: if you want a broad training opt-out, CCBot is one line among several.
Common Crawl links an opt-out registry from its CCBot page. We couldn't confirm how that registry works, so check the CCBot page for details before relying on it.
How to block or throttle CCBot
User-agent: CCBot
Disallow: /
Common Crawl's FAQ says CCBot rechecks robots.txt periodically, so a new rule takes effect without you contacting anyone. If load is the problem rather than reuse, keep it allowed and add a delay:
User-agent: CCBot
Crawl-delay: 10
CCBot also honors nofollow on links and reads Sitemaps you list in robots.txt. Each host and subdomain needs its own robots.txt. The robots.txt checker shows whether your file blocks CCBot.
Verifying a visit
The documented user agent is CCBot/2.0 (https://commoncrawl.org/faq/). Because Common Crawl warns that other bots impersonate it, check the source IP against the ccbot.json ranges. For IPv4 addresses, a reverse lookup should return a hostname such as 18-97-14-84.crawl.commoncrawl.org, and a forward lookup of that name should return the original IP. IPv6 addresses don't have reverse DNS yet, so rely on the IP list for those.
Questions
- Does blocking CCBot remove my pages from existing Common Crawl data?
- No. A robots.txt rule only affects future crawls. Snapshots Common Crawl has already published stay public, along with any datasets others built from them.
- Is CCBot an AI training crawler?
- Common Crawl describes its data as provided for research and analysis and doesn't say it crawls for AI training. Our directory lists it under training because its corpus is widely used to build AI training datasets; that label is our classification, not Common Crawl's.
- Can I slow CCBot down instead of blocking it?
- Yes. Common Crawl's FAQ says CCBot obeys Crawl-delay, so a CCBot group with a line such as "Crawl-delay: 10" reduces its request rate without opting you out.
- How do I tell real CCBot from impostors?
- Common Crawl warns that some crawlers falsely identify as CCBot. Check the source address against its published ccbot.json IP ranges, and for IPv4 run a reverse DNS lookup (expect a crawl.commoncrawl.org hostname) followed by a forward lookup that returns the same IP.
- Does blocking CCBot hurt my search rankings?
- No. CCBot doesn't feed a search engine. Google, Bing and AI search tools use their own crawlers, such as Googlebot, Bingbot and OAI-SearchBot.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.