TikTokSpider
ByteDance's second crawler token. Blocking Bytespider alone leaves this one running.
Reference reviewed by Sona
The operator publishes little or no documentation for this token. Details here come from community crawler registries and observed behavior, not vendor confirmation.
| Operator | ByteDance |
|---|---|
| Powers | TikTok AI feature training |
| Purpose | Model training |
| User-agent token | TikTokSpider |
| Respects robots.txt | Inconsistent |
TikTokSpider is a ByteDance user-agent observed alongside Bytespider, associated with data collection for AI features in TikTok and ByteDance's other products. Its existence is the practical point: most published robots.txt templates name Bytespider and stop, which leaves ByteDance's second token untouched.
Documentation is essentially nonexistent - no crawler page, no purpose statement, no compliance policy, no IP feed. Community registries record its robots.txt compliance as unclear, which is marginally better than the flat "No" recorded against Bytespider, but only because less has been observed rather than because behavior is better.
Given the sibling's record, plan for the same posture. If you have decided to keep ByteDance out, write rules for both tokens and enforce at the CDN or WAF, because a robots.txt Disallow is unlikely to be the mechanism that does the work.
The reverse also holds and is easy to miss: sites that allow Bytespider deliberately, having weighed the crawl volume, should include TikTokSpider in that decision rather than leaving it to a wildcard they may not have thought about.
How TikTokSpider behaves
- Routinely omitted from robots.txt files that cover Bytespider, leaving a gap in ByteDance blocks.
- No operator documentation, purpose statement, or compliance policy.
- Community registries list robots.txt compliance as unclear.
- No IP feed or reverse-DNS convention, so the user-agent is unverifiable.
Allow TikTokSpider
You want your content represented in ByteDance's models and TikTok's AI features, and you have accepted the crawl volume.
User-agent: TikTokSpider Allow: /
Block TikTokSpider
Complete a ByteDance block that a Bytespider-only rule leaves half-done - and enforce it at the edge.
User-agent: TikTokSpider Disallow: /
Heads up: TikTokSpider does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
How to verify TikTokSpider
No published verification method
ByteDance publishes no IP range feed or reverse-DNS convention for either of its tokens, so TikTokSpider traffic cannot be authenticated. As with Bytespider, the combination of a recognizable name and no verification makes it usable cover for unrelated scrapers. Edge rules have to key on the user-agent string and on request behavior.
Check an IP against this botCommonly confused with TikTokSpider
The primary ByteDance crawler and the one everybody blocks. These two need rules together - covering only Bytespider is the most common ByteDance mistake.
TikTokSpider FAQs
I already block Bytespider. Do I need this rule too?
Yes. They are separate tokens with separate robots.txt groups, and a Bytespider-only rule leaves TikTokSpider governed by your wildcard. Covering just one is the most common gap in ByteDance blocks.
Does TikTokSpider respect robots.txt?
Unclear. Community registries record it as unclear rather than compliant, and given that its sibling Bytespider is recorded as flatly not respecting robots.txt, planning for edge enforcement is the safer assumption.
What does TikTokSpider collect?
Community sources associate it with data collection for AI features in TikTok and ByteDance's other products. ByteDance publishes no purpose statement, so the specifics are not documented.
How do I block it effectively?
Filter the user-agent at your CDN or WAF alongside Bytespider, and consider rate limiting. With no published IP feed, rules have to match the user-agent pattern and crawl behavior rather than a verified identity.
Can TikTokSpider read your page right now?
Test any URL and see exactly what AI crawlers receive.