CCBot
Common Crawl's nonprofit crawler. Its open dataset is a starting point for a large share of LLM training pipelines.
Reference reviewed by Sona
| Operator | Common Crawl Foundation |
|---|---|
| Powers | The open Common Crawl dataset used across the AI industry |
| Purpose | Model training |
| User-agent token | CCBot |
| Respects robots.txt | Yes |
CCBot builds Common Crawl, a free and openly available archive of the web maintained by a nonprofit. The crawler itself trains nothing - but because the resulting corpus is public and permanently downloadable, it has become one of the most reused training inputs in the industry.
That indirection is what makes CCBot the highest-leverage single decision in this directory. Blocking GPTBot stops one company's crawl; blocking CCBot reduces your footprint in an unknown number of downstream training runs at labs you will never hear from. The corresponding weakness is symmetric: allowing GPTBot but blocking CCBot is a narrower opt-out than it looks, and blocking GPTBot while allowing CCBot is close to no opt-out at all.
Common Crawl publishes snapshots on a roughly monthly cadence, and the archive is cumulative and immutable. A Disallow added today keeps you out of future crawls; every snapshot already released remains downloadable with your content in it. Unlike a vendor's training set, there is no operator who can be asked to delete it.
Common Crawl explicitly warns that other crawlers falsely identify themselves as CCBot - a predictable consequence of being a well-known, widely allowed nonprofit user-agent. Verify before you treat CCBot traffic as benign.
How CCBot behaves
- Crawls in monthly-ish snapshot campaigns rather than continuously, so traffic arrives in distinct waves.
- Frequently impersonated - Common Crawl documents the problem directly and publishes IP ranges to counter it.
- Does not render JavaScript; the archive stores the raw server response.
- Past snapshots are immutable and remain publicly downloadable, so a block is forward-looking only.
Full user-agent string
CCBot/2.0 (https://commoncrawl.org/faq/)
Allow CCBot
Maximum reach: your content is available to academic researchers and to the many models trained on the open corpus, at no bandwidth cost beyond the crawl.
User-agent: CCBot Allow: /
Block CCBot
One rule that reduces your exposure across many downstream training pipelines at once - the broadest single training opt-out available.
User-agent: CCBot Disallow: /
How to verify CCBot
Reverse DNS + published IP ranges
CCBot supports both checks, which is useful because it is one of the most impersonated user-agents on the list. Reverse DNS on genuine traffic resolves under crawl.commoncrawl.org, and Common Crawl also publishes its IPv4 and IPv6 ranges as JSON at index.commoncrawl.org/ccbot.json. Common Crawl's own documentation raises the impersonation problem unprompted, so treat an unverified CCBot hit as an unknown scraper rather than a nonprofit archive crawl.
Documented reverse-DNS suffixes: .crawl.commoncrawl.org
Commonly confused with CCBot
The pairing that undermines most training opt-outs: blocking GPTBot while allowing CCBot leaves your content in an open corpus that OpenAI and others can still use.
Both feed openly available research corpora, but AI2Bot crawls for Ai2's own open models while CCBot builds a general-purpose public archive.
Webz.io also supplies training data to third parties, but commercially and under license. Common Crawl's dataset is free to anyone.
CCBot FAQs
Why is CCBot considered the most important AI training block?
Because its output is an open dataset that many labs train on. Blocking one company's crawler affects one company; blocking CCBot reduces your presence in a corpus used across the industry, including by labs that never crawl you directly.
Does blocking CCBot remove my content from existing Common Crawl data?
No. Published snapshots are immutable and stay publicly downloadable. A Disallow keeps you out of future crawls only, and unlike a vendor training set there is no operator who can delete the archived copies.
How do I tell real CCBot from an impersonator?
Reverse DNS on genuine traffic resolves under crawl.commoncrawl.org, and the published ranges are at index.commoncrawl.org/ccbot.json. Common Crawl warns explicitly that other crawlers falsely claim its user-agent, so the check is worth running.
Is Common Crawl an AI company?
No, it is a nonprofit maintaining an open web archive for research and analysis. The AI relevance is downstream: because the corpus is free and public, it has become a common training input for others.
Can CCBot read your page right now?
Test any URL and see exactly what AI crawlers receive.