Webzio-Extended
A robots.txt opt-out token for Webz.io, whose web datasets (formerly omgili) are licensed to AI companies for training.
Reference reviewed by Sona
| Operator | Webz.io |
|---|---|
| Powers | Webz.io datasets licensed for AI training |
| Purpose | Training opt-out token |
| User-agent token | Webzio-Extended |
| Respects robots.txt | Yes |
Webz.io crawls the web (historically under the omgili/omgilibot user-agents) and licenses the resulting datasets to customers - including AI companies buying training data. Webzio-Extended is the robots.txt token that specifically opts your content out of AI-training use of those datasets.
Like Google-Extended and Applebot-Extended, it is a control token rather than a bot that sends distinct traffic - set a rule for it alongside any omgilibot rules.
Allow Webzio-Extended
Your content can appear in commercial web datasets that many AI builders train and build on.
User-agent: Webzio-Extended Allow: /
Block Webzio-Extended
One rule keeps your content out of training datasets resold to multiple AI companies at once.
User-agent: Webzio-Extended Disallow: /
Can Webzio-Extended read your page right now?
Test any URL and see exactly what AI crawlers receive.