TikTokSpider
ByteDance's second crawler token, associated with training AI features in TikTok - sibling of the aggressive Bytespider.
Reference reviewed by Sona
| Operator | ByteDance |
|---|---|
| Powers | TikTok AI features training |
| Purpose | Model training |
| User-agent token | TikTokSpider |
| Respects robots.txt | Inconsistent |
TikTokSpider is a ByteDance user-agent observed alongside Bytespider, associated with data collection for AI features in TikTok and ByteDance's other products.
Like Bytespider, ByteDance publishes little documentation for it, and its robots.txt compliance is inconsistent - operators who want it gone typically enforce blocks at the server or firewall level. Set rules for both ByteDance tokens together.
Allow TikTokSpider
You want your content represented in ByteDance's AI models and TikTok's AI features.
User-agent: TikTokSpider Allow: /
Block TikTokSpider
Reduce crawl load and keep content out of ByteDance training - and don't rely on robots.txt alone.
User-agent: TikTokSpider Disallow: /
Heads up: TikTokSpider does not reliably honor robots.txt. To enforce a block, combine the rule above with server- or firewall-level filtering of the user-agent.
Can TikTokSpider read your page right now?
Test any URL and see exactly what AI crawlers receive.