GPTBot
OpenAI's training crawler. Allowing it lets your content be used to train future GPT models.
Reference reviewed by Sona
| Operator | OpenAI |
|---|---|
| Powers | Training data for OpenAI's GPT models |
| Purpose | Model training |
| User-agent token | GPTBot |
| Respects robots.txt | Yes |
GPTBot is the crawler OpenAI uses to gather publicly available web content for training its foundation models. OpenAI's crawler documentation describes its job as crawling content "that may be used in training our generative AI foundation models" - bulk collection, not real-time retrieval for a specific question.
The distinction matters more than most robots.txt decisions on this list, because GPTBot is the only OpenAI crawler whose output is training data. OpenAI runs four separate tokens - GPTBot for training, OAI-SearchBot for the ChatGPT search index, ChatGPT-User for live user-triggered fetches, and OAI-AdsBot for ad safety review - and each reads a different robots.txt group. A blanket disallow aimed at "OpenAI" that only names GPTBot leaves the other three untouched.
GPTBot obeys robots.txt. If you disallow it, OpenAI states the pages will be excluded from future training sets. What a disallow cannot do is claw back what has already been collected, and it has no effect on content OpenAI reaches through third-party datasets - Common Crawl being the significant one, since CCBot's corpus is a separate pipeline with its own token. Blocking GPTBot while leaving CCBot allowed is a common and mostly self-defeating combination.
The user-agent carries a version segment (GPTBot/1.4 at the time of writing) that increments as OpenAI updates the crawler. Match on the GPTBot token rather than the full string, or your rules will silently stop applying after the next bump.
How GPTBot behaves
- Bulk crawler: expect broad, sustained coverage of your URL space rather than the single-page hits a user-triggered fetcher makes.
- Does not execute JavaScript for training collection, so content that only exists after client-side hydration is largely invisible to it.
- Uses a versioned user-agent (GPTBot/1.4) that changes on crawler updates - match the bare token, not the full string.
- Reads only the GPTBot robots.txt group. Rules written for a generic wildcard apply only if no GPTBot group exists.
Full user-agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Allow GPTBot
Your content can influence how GPT models describe your brand, products, and domain from their trained knowledge - the part of ChatGPT that answers without searching.
User-agent: GPTBot Allow: /
Block GPTBot
You don't want your content used as training data, for licensing, copyright, or competitive reasons - and you accept that this does not affect whether ChatGPT can cite you.
User-agent: GPTBot Disallow: /
How to verify GPTBot
Published IP range feed
OpenAI publishes GPTBot's egress ranges as JSON at openai.com/gptbot.json, structured as a `prefixes` array of ipv4Prefix/ipv6Prefix entries. Because it is a bulk crawler on stable infrastructure, the IP-range check is reliable here - match the request IP against the feed and refresh your copy on a schedule rather than per request. There is no documented reverse-DNS hostname for GPTBot, so the feed is the only published check.
Check an IP against this botCommonly confused with GPTBot
GPTBot feeds model training; OAI-SearchBot builds the index that lets ChatGPT cite and link you. Blocking GPTBot does not remove you from ChatGPT search results.
GPTBot crawls in bulk on OpenAI's schedule; ChatGPT-User fetches one page because a person asked ChatGPT to open it.
Blocking GPTBot but allowing CCBot still routes your content into OpenAI training indirectly, since Common Crawl's open dataset is a widely used training input.
GPTBot FAQs
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only governs training. Whether ChatGPT can find, cite, and link your pages is controlled by OAI-SearchBot, and whether it can open a page a user names is controlled by ChatGPT-User. Blocking GPTBot alone leaves both of those intact.
Does disallowing GPTBot remove content already used in training?
No. OpenAI describes the opt-out as applying to future training sets. Content already collected, and content reachable through third-party corpora such as Common Crawl, is not retroactively withdrawn by a robots.txt change.
Should I match on GPTBot or on the full user-agent string?
Match the GPTBot token. The full string contains a version number (currently GPTBot/1.4) that OpenAI increments, so a rule pinned to the whole string will quietly stop matching after an update.
How do I confirm traffic claiming to be GPTBot is genuine?
Check the source IP against openai.com/gptbot.json. The user-agent string alone is trivially forged, and GPTBot has no documented reverse-DNS hostname, so the published IP feed is the only real check.
Can GPTBot read your page right now?
Test any URL and see exactly what AI crawlers receive.