Diffbot · Other crawling
Diffbot: the Knowledge Graph crawler, and its robots.txt limits
Diffbot is the crawler of Diffbot, a web data company: it does general, proactive web crawling to build a general search engine, the base for Diffbot's Knowledge Graph and data products. Diffbot says it does not crawl pages to train generative AI foundation models. Its crawls follow robots.txt by default, but Diffbot says it may ignore or override robots.txt in specific cases, usually under an agreement with the site.
- robots.txt token
Diffbot- User agent
- Not published by the operator
- Operator
- Diffbot
- Type
- Other crawling
- What it feeds
- Diffbot's Knowledge Graph and web data products, built from general, proactive web crawling for a general search engine.
- Follows robots.txt
- Partly. Diffbot says its crawls adhere to robots.txt Disallow and Crawl-delay by default, but robots.txt may be ignored or overridden in specific cases, typically under a partnership or agreement with the site. Customers running Crawlbot can configure robots.txt adherence, which is on by default.
- How to verify it
- No published method
Should you block Diffbot?
Blocking Diffbot takes your site out of future Diffbot crawls, so it drops out of the Knowledge Graph and crawl-based data products that third parties use for enrichment, monitoring and AI grounding. It isn't a training opt-out, because Diffbot says it doesn't crawl for foundation-model training. It also may not stop customer crawls, which can use their own user agent.
User-agent: Diffbot
Disallow: /
User-agent: Diffbot
Allow: /
Add these lines to /robots.txt at your site root. A group naming Diffbot replaces your User-agent: * rules for it, so repeat any paths you also want it to skip.
Live check
Is Diffbot allowed on your site?
Diffbot's two tokens, and customer traffic
Diffbot's robots.txt FAQ describes more than one kind of traffic, and only some of it carries the Diffbot name:
| Traffic | Identifies as | What it is |
|---|---|---|
| Diffbot | Diffbot token | General, proactive crawling for Diffbot's general search engine and Knowledge Graph |
| Diffbot-User | Diffbot-User token | Requests made on behalf of a human user browsing a URL with Diffbot software |
| Customer crawls (Extract, Crawlbot) | The customer's own user agent, as Diffbot instructs | Crawls a customer configures; robots.txt adherence is on by default but configurable |
We class Diffbot as a data provider rather than a search or training crawler: it builds a knowledge graph that other companies buy, rather than a consumer search engine like Bingbot or a training crawler like CCBot. That label is ours.
What blocking it does, and what it doesn't
A disallow for Diffbot keeps your pages out of future default crawls, so they stop flowing into the Knowledge Graph and the datasets built from it. Three limits apply:
- It's not absolute. Diffbot says robots.txt may be ignored or overridden in specific cases, typically under a partnership or agreement with the site. That is why this directory marks it "partial".
- It doesn't cover customers. Crawlbot and Extract customers are told to use their own user agent, so their crawls won't match a Diffbot rule, and a customer can turn off robots.txt adherence in Crawlbot.
- It isn't a training opt-out. Diffbot says it doesn't crawl for generative AI foundation-model training, so block GPTBot and similar tokens for that.
How to block it
User-agent: Diffbot
Disallow: /
User-agent: Diffbot-User
Disallow: /
Drop the Diffbot-User group if you only want to stop proactive crawling and still let people use Diffbot software on your URLs. If load is the concern, a Crawl-delay line in the Diffbot group is honored by default. Each subdomain needs its own robots.txt; the robots.txt checker shows how your file treats both tokens.
Verifying a visit
Diffbot publishes no full user-agent strings, IP ranges or reverse-DNS method, so there's no way to prove a request is Diffbot. Match on the token in the user agent. Diffbot's old docs.diffbot.com addresses now redirect to diffbot.com/docs, so update any bookmarks or links to its documentation.
Questions
- Does Diffbot train AI models on my content?
- Diffbot's docs say it does not crawl pages for training generative AI foundation models. Its data does go to customers, who may use it for AI grounding and other purposes Diffbot's docs don't detail.
- What is Diffbot-User?
- A second token Diffbot documents for requests made on behalf of a human user browsing a URL with Diffbot software. It's separate from the Diffbot crawler token, so name both in robots.txt if you want to cover both.
- Why do I see Diffbot-powered requests that don't say Diffbot?
- Diffbot tells customers using Extract and Crawlbot to set their own organization's user agent, so traffic a customer drives through Diffbot may not identify as Diffbot at all. A Diffbot robots.txt rule won't match it.
- Does Diffbot honor Crawl-delay?
- Yes, by default. Diffbot says its crawls adhere to robots.txt instructions, naming both disallow and crawl-delay, unless a specific case such as an agreement with the site overrides them.
- How do I verify a Diffbot visit?
- Diffbot's docs give no full user-agent string, IP ranges or verification method, so a request can't be independently confirmed. Match on the Diffbot token and treat it as a claim.
Related crawlers
Sources
Checked against these pages on 2026-10-18. Operators change their crawlers; check the source before relying on a detail.