Named AI bot rules in robots.txt

What is measured

The scanner reads the already-fetched /robots.txt and looks for groups whose user-agent line names an AI crawler explicitly (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent) with Allow or Disallow rules of their own. It passes with at least one such named group carrying a rule, returns partial when robots.txt exists but AI names appear only in comments or the only rules sit in the User-agent: * group, and fails when /robots.txt is missing, empty or HTML.

Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

              DOMAIN="example.com"
curl -sSL "https://$DOMAIN/robots.txt" | grep -Ei 'user-agent:[[:space:]]*(gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|google-extended|perplexitybot|perplexity-user|ccbot|bytespider|applebot-extended|meta-externalagent)' | sort -u
            

The fix

This is the artifact to ship. It is the complete change, not a description of one.

              # robots.txt - https://example.com/robots.txt
# Answer engines are welcome and named explicitly.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /

# Training crawlers are named and refused.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
            

Standards referenced

• RFC 9309

Related checks