Bytespider
ByteDance's training crawler. Widely documented as not honoring robots.txt, and notorious for crawl volume.
Reference reviewed by Sona
The operator publishes little or no documentation for this token. Details here come from community crawler registries and observed behavior, not vendor confirmation.
| Operator | ByteDance |
|---|---|
| Powers | ByteDance and TikTok AI model training |
| Purpose | Model training |
| User-agent token | Bytespider |
| Respects robots.txt | Inconsistent |
Bytespider collects web data for ByteDance, the company behind TikTok, to train its AI models. It is the entry on this list with the worst compliance reputation, and the assessment is not a matter of interpretation: the community-maintained ai.robots.txt registry records its robots.txt compliance as a flat "No" rather than the "unclear" it assigns to most thinly documented crawlers.
Volume is the other half of the problem. Bytespider is routinely reported among the heaviest AI crawlers by request count, and unlike the well-behaved bots it offers no working throttle - no Crawl-delay support, no published guidance, no operator contact for rate complaints. For smaller sites it is a hosting-cost issue before it is a content-licensing issue.
ByteDance publishes no crawler documentation page, no IP range feed, and no verification mechanism. That absence has a practical consequence beyond trust: you cannot distinguish genuine Bytespider traffic from anything else sending that user-agent, and because the name is widely blocked, it is also widely borrowed by scrapers hoping to be mistaken for something identifiable.
The upshot is that this is one of the few bots where robots.txt is close to pointless. Sites that want Bytespider gone block it at the CDN or WAF by user-agent pattern, and often rate-limit the originating networks as well. A Disallow line is worth adding for the record, but do not expect it to be what stops the traffic.
How Bytespider behaves
- Documented by the ai.robots.txt registry as not respecting robots.txt - an unambiguous "No", not an "unclear".
- Consistently reported among the highest-volume AI crawlers by request count.
- No Crawl-delay support, no published documentation, and no operator channel for rate complaints.
- No IP feed or rDNS convention, so the user-agent is unverifiable and frequently borrowed by other scrapers.
Allow Bytespider
You want your content represented in ByteDance's models - and you are prepared to absorb crawl volume you cannot throttle.
User-agent: Bytespider Allow: /
Block Bytespider
Cut significant server load and keep content out of ByteDance training - but enforce it at the edge, because robots.txt is documented as not honored.
User-agent: Bytespider Disallow: /
Heads up: Bytespider does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
How to verify Bytespider
No published verification method
ByteDance publishes no IP range feed and no reverse-DNS convention, so genuine Bytespider traffic cannot be distinguished from traffic impersonating it. This cuts both ways in practice: a widely blocked, unverifiable name is attractive cover for unrelated scrapers, so some share of your Bytespider log lines is probably not ByteDance at all. With no way to tell, edge rules here have to key on the user-agent pattern and on behavior - request rate, path fan-out - rather than on identity.
Check an IP against this botCommonly confused with Bytespider
ByteDance's other token, seen alongside this one. Write rules for both - blocking only Bytespider leaves the sibling untouched.
Both are training crawlers, but GPTBot honors robots.txt and publishes verifiable IP ranges. Treating them the same in your enforcement plan will underprotect you here.
Bytespider FAQs
Does Bytespider respect robots.txt?
No. The ai.robots.txt registry records its compliance as "No" outright, and operator reports of it crawling disallowed paths are common. A Disallow line documents your intent but should not be relied on to stop it.
How do I actually block Bytespider?
Filter the user-agent at your CDN or WAF, and consider rate-limiting the source networks as well. Since ByteDance publishes no IP feed, edge rules have to match on the user-agent pattern and on behavior rather than on verified identity.
Why is Bytespider using so much of my bandwidth?
High request volume is its documented characteristic, and there is no working throttle - it does not support Crawl-delay and ByteDance publishes no rate guidance or contact for complaints. For smaller sites this is usually a hosting-cost problem first.
Can I verify that Bytespider traffic is really ByteDance?
No. There is no published IP range feed and no reverse-DNS convention. Because the name is widely blocked and unverifiable, unrelated scrapers use it as cover, so some of what you log as Bytespider is likely something else.
Can Bytespider read your page right now?
Test any URL and see exactly what AI crawlers receive.