# ClaudeBot, Claude-User and Claude-SearchBot: what each Anthropic user agent does and whether it obeys robots.txt

> Does ClaudeBot respect robots.txt? Per Anthropic, yes. What ClaudeBot, Claude-User and Claude-SearchBot each do and what a Disallow for each one changes.

- Source: /learn/anthropic-crawlers-claudebot-claude-user-claude-searchbot
- Updated: 2026-10-27
- Reading time: 6 minutes
- Tags: Anthropic, ClaudeBot, robots.txt, AI crawlers

---
Yes. Anthropic's help article says the 3 user agents it names respect robots.txt. Each has its own robots.txt token, so a Disallow for ClaudeBot opts out of training and leaves search and user questions alone.

ClaudeBot collects web content that could train Anthropic's models. Claude-SearchBot improves the quality of search results. Claude-User retrieves a page when a person asks Claude a question that needs it. Anthropic also supports the non-standard `Crawl-delay` line, and warns that blocking by IP address can defeat an opt-out.

## What do ClaudeBot, Claude-User and Claude-SearchBot do?

This is Anthropic's [help article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) as read on 27 October 2026. Its markup dates the last change to 7 April 2026.

| Token | Used for | What disabling it does, per Anthropic |
| --- | --- | --- |
| `ClaudeBot` | Collecting web content that could contribute to training Anthropic's generative AI models | A restriction signals that the site's future materials should be excluded from training datasets |
| `Claude-User` | Accessing websites when a person asks Claude a question | Claude cannot retrieve your content in response to a user query, which may reduce visibility for user-directed web search |
| `Claude-SearchBot` | Improving search result quality by analysing online content | Your content is not indexed for search, which may reduce visibility and accuracy in user search results |

The article gives no user-agent strings and no version numbers. Match on the token. It also names no other token, so a rule for `anthropic-ai` governs nothing that Anthropic documents today.

## Does ClaudeBot respect robots.txt?

Anthropic says it does. The article states that its user agents “respect ‘do not crawl’ signals” by honouring robots.txt directives. It adds that they respect anti-circumvention technology, and gives the example that they will not try to bypass CAPTCHAs.

The statement covers all 3 tokens, because the article introduces them as the 3 Anthropic uses. That includes `Claude-User`.

This is the operator's promise, and the file cannot enforce it. [RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html) says its rules are not a form of access authorization. To check the promise, look for ClaudeBot in your own log on a path you disallow:

```sh
grep 'ClaudeBot' access.log | grep -E '"GET /account/'
```

The article does not say how often its user agents re-read robots.txt. RFC 9309 §2.4 says a crawler should not use a cached copy for more than 24 hours, unless the file is unreachable. OpenAI and Meta both quote 24 hours as well. Plan for a day before a change takes hold.

## Does Claude-User follow robots.txt too?

Anthropic says it does, and it frames the token as a control. The article says Claude-User lets site owners “control which sites can be accessed” through these user-initiated requests.

That differs from other operators' wording for the same kind of fetch. OpenAI says robots.txt rules “may not apply” to ChatGPT-User. Google says its user-triggered fetchers generally ignore robots.txt rules, and Meta says its user-request fetcher may bypass them. [The OpenAI article](/learn/openai-crawlers-gptbot-oai-searchbot-chatgpt-user) has the exact wording.

A Disallow for Claude-User has a cost. A person asks Claude about your page, and Claude cannot retrieve it. Anthropic's own warning is that this may reduce your site's visibility for user-directed web search.

## How do you block ClaudeBot but keep Claude-SearchBot?

The [AI robots.txt generator](/tools/ai-robots-txt-generator) writes the blocking group for the crawlers you choose.

Write a group for each decision. The first group refuses training. The second keeps search and user questions on the rules everyone else gets.

```txt
# Training: refused
User-agent: ClaudeBot
Disallow: /

# Search and user questions: allowed, minus private paths
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /account/
Allow: /

# Everyone else
User-agent: *
Disallow: /account/
Allow: /
```

Three details decide whether this works.

- **A named group replaces `*`.** Under RFC 9309, a crawler that finds its own group stops reading the `*` group. Repeat any path rule you still want, as the second group does with `/account/`.
- **One file per host.** Anthropic asks you to add the rule for every subdomain you want to opt out. RFC 9309 puts the file at the authority it governs, so `example.com/robots.txt` does not cover `docs.example.com`.
- **The opt-out looks forward.** Anthropic's wording is that a restriction signals “future materials” should be excluded. The article says nothing about content collected earlier.

[AI crawler user agents](/learn/ai-crawler-user-agents) shows how a crawler picks its group, and [How to block AI crawlers](/learn/ai-crawler-blocking) covers the other layers.

## How do you slow ClaudeBot down?

Anthropic supports a `Crawl-delay` line and gives this example:

```txt
User-agent: ClaudeBot
Crawl-delay: 1
```

Anthropic calls the line non-standard, and it is. RFC 9309 does not mention `Crawl-delay`, so a line that Anthropic honours is not a rule every crawler reads. The article also does not define the unit of the number.

The article says nothing about `429` responses or `Retry-After`. [Personal agents with no user agent](/learn/personal-ai-agents-no-user-agent) explains those two for traffic you cannot name.

## Why not block Anthropic by IP address?

Because the opt-out lives in robots.txt. Anthropic says blocking the addresses its crawlers use “may not work correctly or persistently guarantee an opt-out”. A block can stop the crawler from reading your robots.txt at all.

Anthropic links an [IP list](https://claude.com/crawling/bots.json) for identification. If a crawler's source address is on it, the request comes from Anthropic. The article does not say whether one list covers all 3 tokens. Use it to check a request that claims to be ClaudeBot, not to build a block.

## What do real robots.txt files say about these tokens?

Of the 5 external sites in the stored v0.9 records of 20 October 2026, 4 leave all 3 Claude tokens on `*`. Only [GitHub](/aws/github.com) names one, `ClaudeBot`, with its own group of 165 rules, partly disallowed.

| Site | `ClaudeBot` | `Claude-User` | `Claude-SearchBot` |
| --- | --- | --- | --- |
| [Vercel](/aws/vercel.com) | Follows `*` | Follows `*` | Follows `*` |
| [Resend](/aws/resend.com) | Follows `*` | Follows `*` | Follows `*` |
| [MDN](/aws/developer.mozilla.org) | Follows `*` | Follows `*` | Follows `*` |
| [GitHub](/aws/github.com) | Own group, 165 rules, partly disallowed | Follows `*` | Follows `*` |
| [Cloudflare](/aws/cloudflare.com) | Follows `*` | Follows `*` | Follows `*` |

GitHub and Cloudflare each name `anthropic-ai`, the token Anthropic's article does not list. Cloudflare's group for it has 1 rule, which allows. A rule written for that token is aimed at a name Anthropic's current page does not document.

This site's own file shows the fall-through. Read on 27 October 2026, it names `ClaudeBot` and `Claude-User`, and has no group for `Claude-SearchBot`:

```sh
curl -sS https://agenticwebscore.com/robots.txt | grep -i -A3 -E '^user-agent: (claudebot|claude-user|claude-searchbot)'
```

```txt
User-agent: ClaudeBot
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
--
User-agent: Claude-User
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
```

`Claude-SearchBot` therefore follows the `*` group, which holds the same 3 lines. Naming it would state the decision on the record without changing what it may fetch. The [AI crawler policy check](/checks/ai-rules) reads the governing group for each of 18 tokens. To see which group each Claude token gets on your site, [scan your site](/scan).