AI crawler list 2026: robots.txt tokens, who runs each and where they publish IP ranges

An AI crawler token is the name a robots.txt group uses to address one crawler, such as GPTBot or ClaudeBot. The 18 tokens below do 3 different jobs: collecting training data, building a search index, and fetching a page when a person asks. Their operators also differ on robots.txt. Anthropic and Amazon say their crawlers honour it, and OpenAI and Perplexity say their user-initiated fetchers may not follow it.

The page also covers tokens outside the 18, such as Google-Agent, and where each operator publishes its IP ranges.

The 18 tokens and what each one does

This is the list the AI crawler policy check resolves. Each row comes from the operator's own page, read on 27 October 2026.

A token in this table is a request header the client writes. Anyone can send GPTBot, so a rule for it binds the honest operator and nobody else. Cloudflare reported in August 2025 that Perplexity used an undeclared browser user agent after its declared crawlers were blocked, and removed it from its verified list. The post carries no reply from Perplexity. Blocking AI crawlers without locking out agents covers what to do about the rest.

Training, search and user fetches are separate decisions

OpenAI's page says each of its settings is independent. A site can allow OAI-SearchBot to appear in search results, and disallow GPTBot to keep its content out of training. Anthropic splits the same way, with ClaudeBot, Claude-SearchBot and Claude-User. Perplexity says PerplexityBot is not used to crawl content for foundation models.

Two tokens are controls, not crawlers. Google says Google-Extended has no user agent string of its own: existing Google crawlers do the crawling. Disallowing it limits Gemini training and grounding, and it does not affect inclusion in Google Search. Apple says Applebot-Extended does not crawl: a Disallow for it limits how content Applebot already fetched is used to train Apple's models.

Amazonbot and Meta-ExternalAgent mix uses. Amazon says Amazonbot may be used to train Amazon AI models. Meta says Meta-ExternalAgent crawls for training foundation models or for indexing content to improve products.

What the user-initiated fetchers do with robots.txt

OpenAI and Perplexity treat a fetch that a person asked for differently from a crawl. OpenAI says of ChatGPT-User that, because a user starts these actions, robots.txt rules may not apply. Perplexity says Perplexity-User generally ignores robots.txt rules, for the same reason. Google says its user-triggered fetchers, a group that includes Google-Agent, generally ignore robots.txt rules too.

Anthropic differs from OpenAI and Perplexity here. Its page says Anthropic's crawlers honour robots.txt, and it lists Claude-User among them. Mistral AI says its MistralAI-User token governs which sites user requests can be made to.

So a Disallow for ChatGPT-User or Perplexity-User is a request. To refuse those fetches outright, enforce it at the edge, and accept that you also refuse the person who asked.

Tokens with no current documentation

Three of the 18 have no operator page that describes them today. A rule for one costs a few lines and may govern nothing.

• anthropic-ai: Anthropic's page names ClaudeBot, Claude-User and Claude-SearchBot, and not this token.
• cohere-ai: Cohere's page says it does not use crawlers to collect web content at this time, and its table of crawlers is empty.
• Bytespider: no English page from ByteDance documents it. Its Toutiao webmaster platform loaded with no readable content on 25 October 2026.

Tokens outside the 18, including Google-Agent and Meta-WebIndexer

The check does not resolve these documented tokens, but robots.txt accepts a rule for any of them.

Meta's Muse has no token to list

Meta's crawler page documents 5 crawlers, and none is named Muse. Search Engine Journal reports that Muse has no published user agent. It also reports that Amazon blocked Muse on 20 September 2026, and that Amazon says Muse does not identify itself.

A robots.txt group needs a name to address, so a rule cannot reach Muse. What a site can do about agents with no user agent covers the options.

IP ranges: where each operator publishes them

An operator's IP list is the second check: compare the request's source address with it. Each URL below comes from the operator's own page, read on 27 October 2026. We did not download the lists.

Google, Apple and Common Crawl also document reverse DNS names for their crawlers. Use a list to identify traffic, not to block it. Anthropic warns that an IP block can stop its crawler reading robots.txt, as the blocking guide explains.

How RFC 9309 picks the group a crawler obeys

RFC 9309 has a crawler find the group that names its product token, with case-insensitive matching. It obeys the * group only “if no matching group exists”. So a named group replaces the * group for that crawler. Google's robots.txt page says the same.

A group can name several tokens, and groups that name the same token are merged.

The trap looks like this:

User-agent: *
Disallow: /account/
Disallow: /api/

User-agent: GPTBot
Allow: /

GPTBot may now fetch /account/ and /api/, because its own group has no Disallow. Repeat every * rule you still want in each named group.

Example robots.txt policy

Refuse training, and leave search and user fetches on the rules everyone else gets:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /account/
Allow: /

Blocking AI crawlers without locking out agents has the full file. Add a Content-Signal line to each group to state the use you allow, as Content-Signal in robots.txt explains.

The AI crawler policy check reads a token as having rules only when its group allows /. A file whose every named group is Disallow: / reads as Not used: the named groups block their crawlers and state nothing else. The second group above is what makes this file state a policy.

How to check what each crawler will read

The AI crawler checker resolves the governing group for all 18 tokens on your site.

Fetch the file the crawler fetches, and read the group for one token:

curl -sS https://agenticwebscore.com/robots.txt | grep -i -A3 '^user-agent: gptbot'

User-agent: GPTBot
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes

This site's file names 17 of the 18 tokens. Claude-SearchBot has no group, so it follows *, which holds the same 3 lines.

In the scan records of 20 October 2026, 3 of the 5 sites name none of the 18 tokens. Every token follows the * group at Vercel, Resend and MDN. GitHub names 6 and disallows Bytespider, and Cloudflare names 7.

The check reports, for each token, whether its governing group allows it, partly disallows it, or disallows it. It earns points in the agent-conventions part and holds no level. To see the group each token gets on your site, scan your site.