ClaudeBot, Claude-User and Claude-SearchBot: what each Anthropic user agent does and whether it obeys robots.txt

Yes. Anthropic's help article says the 3 user agents it names respect robots.txt. Each has its own robots.txt token, so a Disallow for ClaudeBot opts out of training and leaves search and user questions alone.

ClaudeBot collects web content that could train Anthropic's models. Claude-SearchBot improves the quality of search results. Claude-User retrieves a page when a person asks Claude a question that needs it. Anthropic also supports the non-standard Crawl-delay line, and warns that blocking by IP address can defeat an opt-out.

What do ClaudeBot, Claude-User and Claude-SearchBot do?

This is Anthropic's help article as read on 27 October 2026. Its markup dates the last change to 7 April 2026.

The article gives no user-agent strings and no version numbers. Match on the token. It also names no other token, so a rule for anthropic-ai governs nothing that Anthropic documents today.

Does ClaudeBot respect robots.txt?

Anthropic says it does. The article states that its user agents “respect ‘do not crawl’ signals” by honouring robots.txt directives. It adds that they respect anti-circumvention technology, and gives the example that they will not try to bypass CAPTCHAs.

The statement covers all 3 tokens, because the article introduces them as the 3 Anthropic uses. That includes Claude-User.

This is the operator's promise, and the file cannot enforce it. RFC 9309 says its rules are not a form of access authorization. To check the promise, look for ClaudeBot in your own log on a path you disallow:

grep 'ClaudeBot' access.log | grep -E '"GET /account/'

The article does not say how often its user agents re-read robots.txt. RFC 9309 §2.4 says a crawler should not use a cached copy for more than 24 hours, unless the file is unreachable. OpenAI and Meta both quote 24 hours as well. Plan for a day before a change takes hold.

Does Claude-User follow robots.txt too?

Anthropic says it does, and it frames the token as a control. The article says Claude-User lets site owners “control which sites can be accessed” through these user-initiated requests.

That differs from other operators' wording for the same kind of fetch. OpenAI says robots.txt rules “may not apply” to ChatGPT-User. Google says its user-triggered fetchers generally ignore robots.txt rules, and Meta says its user-request fetcher may bypass them. The OpenAI article has the exact wording.

A Disallow for Claude-User has a cost. A person asks Claude about your page, and Claude cannot retrieve it. Anthropic's own warning is that this may reduce your site's visibility for user-directed web search.

How do you block ClaudeBot but keep Claude-SearchBot?

The AI robots.txt generator writes the blocking group for the crawlers you choose.

Write a group for each decision. The first group refuses training. The second keeps search and user questions on the rules everyone else gets.

# Training: refused
User-agent: ClaudeBot
Disallow: /

# Search and user questions: allowed, minus private paths
User-agent: Claude-SearchBot
User-agent: Claude-User
Disallow: /account/
Allow: /

# Everyone else
User-agent: *
Disallow: /account/
Allow: /

Three details decide whether this works.

• A named group replaces *. Under RFC 9309, a crawler that finds its own group stops reading the * group. Repeat any path rule you still want, as the second group does with /account/.
• One file per host. Anthropic asks you to add the rule for every subdomain you want to opt out. RFC 9309 puts the file at the authority it governs, so example.com/robots.txt does not cover docs.example.com.
• The opt-out looks forward. Anthropic's wording is that a restriction signals “future materials” should be excluded. The article says nothing about content collected earlier.

AI crawler user agents shows how a crawler picks its group, and How to block AI crawlers covers the other layers.

How do you slow ClaudeBot down?

Anthropic supports a Crawl-delay line and gives this example:

User-agent: ClaudeBot
Crawl-delay: 1

Anthropic calls the line non-standard, and it is. RFC 9309 does not mention Crawl-delay, so a line that Anthropic honours is not a rule every crawler reads. The article also does not define the unit of the number.

The article says nothing about 429 responses or Retry-After. Personal agents with no user agent explains those two for traffic you cannot name.

Why not block Anthropic by IP address?

Because the opt-out lives in robots.txt. Anthropic says blocking the addresses its crawlers use “may not work correctly or persistently guarantee an opt-out”. A block can stop the crawler from reading your robots.txt at all.

Anthropic links an IP list for identification. If a crawler's source address is on it, the request comes from Anthropic. The article does not say whether one list covers all 3 tokens. Use it to check a request that claims to be ClaudeBot, not to build a block.

What do real robots.txt files say about these tokens?

Of the 5 external sites in the stored v0.9 records of 20 October 2026, 4 leave all 3 Claude tokens on *. Only GitHub names one, ClaudeBot, with its own group of 165 rules, partly disallowed.

GitHub and Cloudflare each name anthropic-ai, the token Anthropic's article does not list. Cloudflare's group for it has 1 rule, which allows. A rule written for that token is aimed at a name Anthropic's current page does not document.

This site's own file shows the fall-through. Read on 27 October 2026, it names ClaudeBot and Claude-User, and has no group for Claude-SearchBot:

curl -sS https://agenticwebscore.com/robots.txt | grep -i -A3 -E '^user-agent: (claudebot|claude-user|claude-searchbot)'

User-agent: ClaudeBot
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes
--
User-agent: Claude-User
Disallow: /api/
Allow: /
Content-Signal: ai-train=no, search=yes, ai-input=yes

Claude-SearchBot therefore follows the * group, which holds the same 3 lines. Naming it would state the decision on the record without changing what it may fetch. The AI crawler policy check reads the governing group for each of 18 tokens. To see which group each Claude token gets on your site, scan your site.