GPTBot vs OAI-SearchBot vs ChatGPT-User: what each OpenAI crawler does and what blocking it changes

GPTBot collects content that may train OpenAI's models. OAI-SearchBot finds pages for ChatGPT search. ChatGPT-User can fetch a page when a person asks ChatGPT, or a custom GPT, a question about it. Each has its own robots.txt token.

OpenAI says the GPTBot and OAI-SearchBot settings are independent. For ChatGPT-User it says robots.txt rules “may not apply”, so a Disallow there is a request and not a block.

What does each OpenAI crawler do?

OpenAI's crawler page lists 4 user agents. The table is what it says about each, read on 27 October 2026.

OpenAI says OAI-AdsBot only visits pages submitted as ads, and that its data is not used to train foundation models. The rest of this page covers the other 3.

What changes if you block GPTBot?

You tell OpenAI not to use your content for training. OpenAI's wording is that disallowing GPTBot “indicates” this, so it is a stated preference. The page does not say what happens to content collected before the Disallow.

OpenAI assigns search to OAI-SearchBot, not to GPTBot. Its own example allows OAI-SearchBot so a site appears in results, and disallows GPTBot to opt out of training. If you allow both, OpenAI says it may use the results of 1 crawl for both uses, to avoid duplicate crawling.

What changes if you block OAI-SearchBot?

Your pages stop appearing in ChatGPT search answers. They can still show up as navigational links. OpenAI says its systems can take about 24 hours to adjust after a robots.txt change.

To appear in search results, OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. A robots.txt that welcomes the crawler does nothing if your firewall refuses its addresses. Check both.

Does ChatGPT-User obey robots.txt?

OpenAI's page gives its answer in one sentence:

Because these actions are initiated by a user, robots.txt rules may not apply.

The same entry says ChatGPT-User does not crawl the web in an automatic fashion. It also says the token plays no part in deciding whether content appears in search. For search opt-outs and automatic crawling, OpenAI points to OAI-SearchBot.

So a Disallow for ChatGPT-User states your preference and promises nothing. To refuse these fetches, enforce it at your edge, using the source address. The person who asked then gets no answer from your page.

OpenAI is not alone in this. Google says its user-triggered fetchers “generally ignore robots.txt rules”, and Meta says its user-request fetcher “may bypass robots.txt rules”. Anthropic says its equivalent, Claude-User, does honour robots.txt, as the Anthropic article explains.

Which robots.txt rules match which goal?

The AI robots.txt generator writes the blocking group for the crawlers you choose.

Each goal below is a rule, plus what OpenAI's page lets you expect from it.

A group that names a token replaces the * group for that token. Under RFC 9309, a crawler that finds its own group stops reading *, so repeat the path rules you still want:

# Training: refused
User-agent: GPTBot
Disallow: /

# ChatGPT search and user fetches: allowed, minus private paths
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
Disallow: /account/
Allow: /

# Everyone else
User-agent: *
Disallow: /account/
Allow: /

AI crawler user agents explains how a crawler picks its group, and How to block AI crawlers covers the trade-offs of each layer.

How do you tell the three apart in your logs?

Match the token in the user-agent string. OpenAI gives these strings and says the version number may change, so match GPTBot, not GPTBot/1.4:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI adds that GPTBot and OAI-SearchBot may carry an extra robots.txt marker in the string when they fetch your robots.txt. The marker separates those requests from page requests when your log omits paths. For OAI-SearchBot, OpenAI shows it as ; robots.txt; between the version and the URL. These commands count each user agent, then the marked requests:

grep -oE 'GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot' access.log | sort | uniq -c | sort -rn
grep -c '; robots.txt;' access.log

A user-agent string is a claim anyone can type. OpenAI publishes a separate IP list for each user agent, so check a request that says ChatGPT-User against chatgpt-user.json, not gptbot.json. A request that names a token from an address outside its list is not from OpenAI.

What do real robots.txt files say about these tokens?

Of the 5 external sites in the stored v0.9 scan records of 20 October 2026, none disallows / for any OpenAI token. Two sites give a token its own group, and 3 give none.

A token that follows * has no rule of its own. A reader cannot tell whether the site chose that or never considered OpenAI. The AI crawler policy check reads the group that governs each of 18 tokens. It counts a token as having rules when that token's own group allows /.

What robots.txt does not cover

OpenAI's page lists no token for an agent that drives a browser for a person. There is no robots.txt line to write for one. Personal agents with no user agent covers what a site can still do.

To see which group governs GPTBot, OAI-SearchBot and ChatGPT-User on your own site, scan your site.