Diffbot
A commercial extraction crawler. Mass crawls honor robots.txt - but customer-requested single URLs may not.
Reference reviewed by Sona
| Operator | Diffbot |
|---|---|
| Powers | Structured web data and a knowledge graph licensed to AI builders |
| Purpose | Model training |
| User-agent token | Diffbot |
| Respects robots.txt | Mostly |
Diffbot machine-reads web pages into structured records and a large knowledge graph, which it licenses to customers including AI companies. Allowing it is therefore an indirect decision: your content propagates into whatever those customers build, and you have no visibility into who they are.
Its robots.txt behavior has a specific and important carve-out. Diffbot's documentation states that its mass crawls respect robots.txt, including Disallow and Crawl-delay - but that extractions of specific URLs requested by customers can be processed even where a block is in place. So the same user-agent honors your rules in one mode and may not in the other, depending on how the request originated.
That makes Diffbot genuinely different from the vendors whose compliance is simply unclear. Here the policy is documented and the exception is documented, so you know exactly what you are accepting: bulk crawling stops if you ask, and targeted per-URL extraction may not.
Diffbot also offers a robots.txt adherence toggle in Crawlbot, on by default, which customers can configure for their own crawls. Version strings vary - Diffbot/1.0 and Diffbot/2.0 both circulate - so match on the Diffbot token rather than a full string.
How Diffbot behaves
- Mass crawls honor robots.txt including Crawl-delay; customer-requested single-URL extractions may proceed despite a block.
- Customers configure their own Crawlbot jobs, with robots.txt adherence toggled on by default.
- Version strings vary between Diffbot/1.0 and Diffbot/2.0 - match the bare token.
- Output is a licensed knowledge graph, so exposure is to Diffbot's customers rather than to Diffbot alone.
Full user-agent string
Mozilla/5.0 (compatible; Diffbot/2.0; +http://www.diffbot.com)
Allow Diffbot
Accurate structured data about your business propagates into the many products and AI systems built on Diffbot's knowledge graph.
User-agent: Diffbot Allow: /
Block Diffbot
You don't want your content extracted and relicensed as structured data - though note customer-requested single-URL extraction may proceed regardless.
User-agent: Diffbot Disallow: /
How to verify Diffbot
No published verification method
Diffbot publishes no IP range feed or reverse-DNS convention. Verification is further complicated by the architecture: customers run their own Crawlbot jobs, so traffic can originate from a range of infrastructure rather than one predictable pool. Match on the Diffbot token and treat version-string variation as expected rather than as a spoofing signal.
Check an IP against this botCommonly confused with Diffbot
Both resell web data commercially. Diffbot sells structured extractions and a knowledge graph; Webz.io sells crawl datasets with AI-training usability tagging.
Both create datasets others train on, but Common Crawl's is free and public while Diffbot's is licensed and commercial.
Diffbot FAQs
Does Diffbot respect robots.txt?
In its mass crawls, yes - Diffbot documents honoring Disallow and Crawl-delay. But it also documents that extractions of specific URLs requested by customers can be processed even where a block exists, so compliance depends on how the request originated.
Who ends up with my content if I allow Diffbot?
Diffbot's licensing customers, which include AI companies using the data for training, grounding, and enrichment. You have no visibility into that list, which is the main argument for treating this as a broader decision than one crawler.
Why does the Diffbot version number keep changing in my logs?
Both Diffbot/1.0 and Diffbot/2.0 circulate, and customer-run Crawlbot jobs can present differently. Match on the Diffbot token rather than pinning a rule to a full user-agent string.
Can I verify Diffbot traffic?
No. There is no published IP feed or reverse-DNS convention, and because customers run their own crawl jobs the traffic can come from varied infrastructure rather than a single predictable pool.
Can Diffbot read your page right now?
Test any URL and see exactly what AI crawlers receive.