# Named AI bot rules in robots.txt

> Named AI bot rules in robots.txt: Naming agent crawlers explicitly is the first step to being visible in agent traffic. Standard: RFC 9309.

- Source: /checks/ai-rules
- Site: AgenticWebScore
- Language: English

---
Naming agent crawlers explicitly is the first step to being visible in agent traffic.

| Property | Value |
| --- | --- |
| Check id | `ai-rules` |
| Dimension | Bot Access Control (18%) |
| Tier | Essential — pool 80 pts |
| Weight inside tier | 2 |
| Impact | High |
| Maturity | Draft — A published specification that is not yet ratified. |
| Conditional | No |
| Publishing skill | `ai-rules` |
| Standards | RFC 9309 |

## What is measured

The scanner reads the already-fetched /robots.txt and looks for groups whose user-agent line names an AI crawler explicitly (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Google-Extended, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent) with Allow or Disallow rules of their own. It passes with at least one such named group carrying a rule, returns partial when robots.txt exists but AI names appear only in comments or the only rules sit in the User-agent: * group, and fails when /robots.txt is missing, empty or HTML.

## Reproduce it yourself

You do not have to take our word for any of this. Run the same measurement against your own domain:

```bash
DOMAIN="example.com"
curl -sSL "https://$DOMAIN/robots.txt" | grep -Ei 'user-agent:[[:space:]]*(gptbot|oai-searchbot|chatgpt-user|claudebot|claude-user|claude-searchbot|google-extended|perplexitybot|perplexity-user|ccbot|bytespider|applebot-extended|meta-externalagent)' | sort -u
```

## The fix

This is the artifact to ship. It is the complete change, not a description of one.

```text
# robots.txt - https://example.com/robots.txt
# Answer engines are welcome and named explicitly.
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /

# Training crawlers are named and refused.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /

User-agent: *
Allow: /
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml
```

> Under RFC 9309 a crawler obeys the group whose product token matches it and only falls back to User-agent: * when no named group matches, so repeat inside each named block every Allow or Disallow you want that crawler to see.

## Standards referenced

- `RFC 9309`

## Related checks

| Check | Tier | Impact | Same skill |
| --- | --- | --- | --- |
| `webbotauth` — Web Bot Auth key directory | Essential | Critical | — |
| `agent-permissions` — Agent permission model (agent-permissions.json) | Essential | Critical | — |
| `ai-policy` — AI usage policy page | Essential | High | — |
| `bot-page` — Crawler and agent policy page | Recommended | Medium | — |
| `security-txt` — security.txt (RFC 9116) | Recommended | Low | — |
| `crawl-delay` — Crawl rate declaration | Emerging | Low | — |
| `ai-txt` — /ai.txt access declaration | Emerging | Low | — |